<!DOCTYPE html>


<html lang="zh-CN">
  

    <head>
      <meta charset="utf-8" />
        
      <meta name="description" content="一个有趣的空间" />
      
      <meta
        name="viewport"
        content="width=device-width, initial-scale=1, maximum-scale=1"
      />
      <title>部署一套完整的企业级K8s集群 |  白糖蘸一点点的空间</title>
  <meta name="generator" content="hexo-theme-ayer">
      
      <link rel="shortcut icon" href="/favicon.ico" />
       
<link rel="stylesheet" href="/dist/main.css">

      <link
        rel="stylesheet"
        href="https://cdn.jsdelivr.net/gh/Shen-Yu/cdn/css/remixicon.min.css"
      />
      
<link rel="stylesheet" href="/css/custom.css">
 
      <script src="https://cdn.jsdelivr.net/npm/pace-js@1.0.2/pace.min.js"></script>
       
 

      <link
        rel="stylesheet"
        href="https://cdn.jsdelivr.net/npm/@sweetalert2/theme-bulma@5.0.1/bulma.min.css"
      />
      <script src="https://cdn.jsdelivr.net/npm/sweetalert2@11.0.19/dist/sweetalert2.min.js"></script>

      <!-- mermaid -->
      
      <style>
        .swal2-styled.swal2-confirm {
          font-size: 1.6rem;
        }
      </style>
    <link rel="alternate" href="/atom.xml" title="白糖蘸一点点的空间" type="application/atom+xml">
</head>
  </html>
</html>


<body>
  <div id="app">
    
      
    <main class="content on">
      <section class="outer">
  <article
  id="post-《二进制方式搭建一个完整K8s集群》v1.20-详细版"
  class="article article-type-post"
  itemscope
  itemprop="blogPost"
  data-scroll-reveal
>
  <div class="article-inner">
    
    <header class="article-header">
       
<h1 class="article-title sea-center" style="border-left:0" itemprop="name">
  部署一套完整的企业级K8s集群
</h1>
 

      
    </header>
     
    <div class="article-meta">
      <a href="/2021/11/16/%E3%80%8A%E4%BA%8C%E8%BF%9B%E5%88%B6%E6%96%B9%E5%BC%8F%E6%90%AD%E5%BB%BA%E4%B8%80%E4%B8%AA%E5%AE%8C%E6%95%B4K8s%E9%9B%86%E7%BE%A4%E3%80%8Bv1.20-%E8%AF%A6%E7%BB%86%E7%89%88/" class="article-date">
  <time datetime="2021-11-16T10:29:00.000Z" itemprop="datePublished">2021-11-16</time>
</a> 
  <div class="article-category">
    <a class="article-category-link" href="/categories/Kubernetes-Docker/">Kubernetes+Docker</a>
  </div>
  
<div class="word_count">
    <span class="post-time">
        <span class="post-meta-item-icon">
            <i class="ri-quill-pen-line"></i>
            <span class="post-meta-item-text"> 字数统计:</span>
            <span class="post-count">11.5k</span>
        </span>
    </span>

    <span class="post-time">
        &nbsp; | &nbsp;
        <span class="post-meta-item-icon">
            <i class="ri-book-open-line"></i>
            <span class="post-meta-item-text"> 阅读时长≈</span>
            <span class="post-count">60 分钟</span>
        </span>
    </span>
</div>
 
    </div>
      
    <div class="tocbot"></div>




  
    <div class="article-entry" itemprop="articleBody">
       
  <h1 id="《部署一套完整的企业级K8s集群》"><a href="#《部署一套完整的企业级K8s集群》" class="headerlink" title="《部署一套完整的企业级K8s集群》"></a>《部署一套完整的企业级K8s集群》</h1><p>v1.24.0-alpha.1 ，二进制方式</p>
<table>
<thead>
<tr>
<th align="left">作者信息</th>
<th align="left">李振良（阿良），微信：k8init</th>
</tr>
</thead>
<tbody><tr>
<td align="left">说明</td>
<td align="left">该文档有导航窗格，方便阅读，如果左侧没有显示，请检查word是否启用。转载请注明作者，拒绝不道德行为！</td>
</tr>
</tbody></table>
<p>本文档是在李振良老师文档基础上做了修改，并按照文档成功搭建了k8s环境，感谢老实的文档，少走了很多弯路</p>
<h1 id="一、前置知识点"><a href="#一、前置知识点" class="headerlink" title="一、前置知识点"></a>一、前置知识点</h1><h2 id="1-1-生产环境部署K8s集群的两种方式"><a href="#1-1-生产环境部署K8s集群的两种方式" class="headerlink" title="1.1 生产环境部署K8s集群的两种方式"></a>1.1 生产环境部署K8s集群的两种方式</h2><ul>
<li>kubeadm<br>Kubeadm是一个K8s部署工具，提供kubeadm init和kubeadm join，用于快速部署Kubernetes集群。  </li>
<li>二进制包<br>从github下载发行版的二进制包，手动部署每个组件，组成Kubernetes集群。  </li>
<li>小结：Kubeadm降低部署门槛，但屏蔽了很多细节，遇到问题很难排查。如果想更容易可控，推荐使用二进制包部署Kubernetes集群，虽然手动部署麻烦点，期间可以学习很多工作原理，也利于后期维护。 </li>
</ul>
<h2 id="1-2-准备环境"><a href="#1-2-准备环境" class="headerlink" title="1.2 准备环境"></a>1.2 准备环境</h2><ul>
<li>服务器要求：<ul>
<li>建议最小硬件配置：2核CPU、2G内存、30G硬盘</li>
<li>服务器最好可以访问外网，会有从网上拉取镜像需求，如果服务器不能上网，需要提前下载对应镜像并导入节点<br>软件环境：</li>
</ul>
</li>
</ul>
<table>
<thead>
<tr>
<th>软件</th>
<th>版本</th>
</tr>
</thead>
<tbody><tr>
<td>操作系统</td>
<td>CentOS7.x_x64 （mini）</td>
</tr>
<tr>
<td>容器引擎</td>
<td>Docker CE 19</td>
</tr>
<tr>
<td>Kubernetes</td>
<td>Kubernetes v1.20</td>
</tr>
</tbody></table>
<p>服务器整体规划：</p>
<table>
<thead>
<tr>
<th>角色</th>
<th>IP</th>
<th>组件</th>
</tr>
</thead>
<tbody><tr>
<td>k8s-master1</td>
<td>192.168.100.61</td>
<td>kube-apiserver，kube-controller-manager，kube-scheduler，kubelet，kube-proxy，docker，etcd，nginx，keepalived</td>
</tr>
<tr>
<td>k8s-master2</td>
<td>192.168.100.64</td>
<td>kube-apiserver，kube-controller-manager，kube-scheduler，kubelet，kube-proxy，docker，nginx，keepalived</td>
</tr>
<tr>
<td>k8s-node1</td>
<td>192.168.100.62</td>
<td>kubelet，kube-proxy，docker，etcd</td>
</tr>
<tr>
<td>k8s-node2</td>
<td>192.168.100.63</td>
<td>kubelet，kube-proxy，docker，etcd</td>
</tr>
<tr>
<td>负载均衡器IP</td>
<td>192.168.100.65 (VIP)</td>
<td></td>
</tr>
</tbody></table>
<p>须知：考虑到有些朋友电脑配置较低，一次性开四台机器会跑不动，所以搭建这套K8s高可用集群分两部分实施，先部署一套单Master架构（3台），再扩容为多Master架构（4台或6台），顺便再熟悉下Master扩容流程。<br>单Master架构图：<br><img src="https://img-blog.csdnimg.cn/df97d916eb1b432483d7b5434674eee8.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5rKn5rW35LmL5beF,size_19,color_FFFFFF,t_70,g_se,x_16" alt="在这里插入图片描述"></p>
<p>单Master服务器规划：</p>
<table>
<thead>
<tr>
<th>角色</th>
<th>IP</th>
<th>组件</th>
</tr>
</thead>
<tbody><tr>
<td>k8s-master</td>
<td>192.168.100.61</td>
<td>kube-apiserver，kube-controller-manager，kube-scheduler，kubelet，kube-proxy，docker，etcd，（3g内存）</td>
</tr>
<tr>
<td>k8s-node1</td>
<td>192.168.100.62</td>
<td>kubelet，kube-proxy，docker，etcd</td>
</tr>
<tr>
<td>k8s-node2</td>
<td>192.168.100.63</td>
<td>kubelet，kube-proxy，docker，etcd</td>
</tr>
</tbody></table>
<h2 id="1-3-操作系统初始化配置"><a href="#1-3-操作系统初始化配置" class="headerlink" title="1.3 操作系统初始化配置"></a>1.3 操作系统初始化配置</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 关闭防火墙 </span></span><br><span class="line">systemctl stop firewalld &amp;&amp; systemctl <span class="built_in">disable</span> firewalld </span><br><span class="line"> </span><br><span class="line"><span class="comment"># 关闭selinux </span></span><br><span class="line">sed -i <span class="string">&#x27;s/enforcing/disabled/&#x27;</span> /etc/selinux/config  <span class="comment"># 永久 </span></span><br><span class="line">setenforce 0  <span class="comment"># 临时 </span></span><br><span class="line"> </span><br><span class="line"><span class="comment"># 关闭swap </span></span><br><span class="line">swapoff -a  <span class="comment"># 临时 </span></span><br><span class="line">sed -ri <span class="string">&#x27;s/.*swap.*/#&amp;/&#x27;</span> /etc/fstab    <span class="comment"># 永久 </span></span><br><span class="line"> </span><br><span class="line"><span class="comment"># 根据规划设置主机名 </span></span><br><span class="line">hostnamectl set-hostname &lt;hostname&gt; </span><br><span class="line"> </span><br><span class="line"><span class="comment"># 在master添加hosts </span></span><br><span class="line">cat &gt;&gt; /etc/hosts &lt;&lt; <span class="string">EOF </span></span><br><span class="line"><span class="string">192.168.100.61 k8s-master1 </span></span><br><span class="line"><span class="string">192.168.100.62 k8s-node1 </span></span><br><span class="line"><span class="string">192.168.100.63 k8s-node2 </span></span><br><span class="line"><span class="string">192.168.100.64 k8s-master2 </span></span><br><span class="line"><span class="string">EOF</span></span><br><span class="line"> </span><br><span class="line"><span class="comment"># 将桥接的IPv4流量传递到iptables的链 </span></span><br><span class="line">cat &gt; /etc/sysctl.d/k8s.conf &lt;&lt; <span class="string">EOF </span></span><br><span class="line"><span class="string">net.bridge.bridge-nf-call-ip6tables = 1 </span></span><br><span class="line"><span class="string">net.bridge.bridge-nf-call-iptables = 1 </span></span><br><span class="line"><span class="string">EOF</span></span><br><span class="line">sysctl --system  <span class="comment"># 生效 </span></span><br><span class="line"> </span><br><span class="line"><span class="comment"># 时间同步 </span></span><br><span class="line">yum install ntpdate -y &amp;&amp; ntpdate time.windows.com</span><br></pre></td></tr></table></figure>

<h1 id="二、部署Etcd集群"><a href="#二、部署Etcd集群" class="headerlink" title="二、部署Etcd集群"></a>二、部署Etcd集群</h1><p>Etcd 是一个分布式键值存储系统，Kubernetes使用Etcd进行数据存储，所以先准备一个Etcd数据库，为解决Etcd单点故障，应采用集群方式部署，这里使用3台组建集群，可容忍1台机器故障，当然，你也可以使用5台组建集群，可容忍2台机器故障。<br>节点名称    | IP<br>————|——-<br>etcd-1    |192.168.100.61<br>etcd-2    |192.168.100.62<br>etcd-3    |192.168.100.63</p>
<p>注：为了节省机器，这里与K8s节点机器复用。也可以独立于k8s集群之外部署，只要apiserver能连接到就行。</p>
<h2 id="2-1-准备cfssl证书生成工具"><a href="#2-1-准备cfssl证书生成工具" class="headerlink" title="2.1 准备cfssl证书生成工具"></a>2.1 准备cfssl证书生成工具</h2><p>cfssl是一个开源的证书管理工具，使用json文件生成证书，相比openssl更方便使用。<br>找任意一台服务器操作，这里用Master节点。</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64</span><br><span class="line">wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64</span><br><span class="line">wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64</span><br><span class="line">chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64</span><br><span class="line">mv cfssl_linux-amd64 /usr/local/bin/cfssl</span><br><span class="line">mv cfssljson_linux-amd64 /usr/local/bin/cfssljson</span><br><span class="line">mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo</span><br></pre></td></tr></table></figure>
<h2 id="2-2-生成Etcd证书"><a href="#2-2-生成Etcd证书" class="headerlink" title="2.2 生成Etcd证书"></a>2.2 生成Etcd证书</h2><h3 id="1-自签证书颁发机构（CA）"><a href="#1-自签证书颁发机构（CA）" class="headerlink" title="1. 自签证书颁发机构（CA）"></a>1. 自签证书颁发机构（CA）</h3><p>创建工作目录：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">mkdir -p ~/TLS/&#123;etcd,k8s&#125;</span><br><span class="line"></span><br><span class="line">cd ~/TLS/etcd</span><br></pre></td></tr></table></figure>
<p>自签CA：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; ca-config.json &lt;&lt; EOF</span><br><span class="line">&#123;</span><br><span class="line">  &quot;signing&quot;: &#123;</span><br><span class="line">    &quot;default&quot;: &#123;</span><br><span class="line">      &quot;expiry&quot;: &quot;87600h&quot;</span><br><span class="line">    &#125;,</span><br><span class="line">    &quot;profiles&quot;: &#123;</span><br><span class="line">      &quot;www&quot;: &#123;</span><br><span class="line">         &quot;expiry&quot;: &quot;87600h&quot;,</span><br><span class="line">         &quot;usages&quot;: [</span><br><span class="line">            &quot;signing&quot;,</span><br><span class="line">            &quot;key encipherment&quot;,</span><br><span class="line">            &quot;server auth&quot;,</span><br><span class="line">            &quot;client auth&quot;</span><br><span class="line">        ]</span><br><span class="line">      &#125;</span><br><span class="line">    &#125;</span><br><span class="line">  &#125;</span><br><span class="line">&#125;</span><br><span class="line">EOF</span><br><span class="line"></span><br><span class="line">cat &gt; ca-csr.json &lt;&lt; EOF</span><br><span class="line">&#123;</span><br><span class="line">    &quot;CN&quot;: &quot;etcd CA&quot;,</span><br><span class="line">    &quot;key&quot;: &#123;</span><br><span class="line">        &quot;algo&quot;: &quot;rsa&quot;,</span><br><span class="line">        &quot;size&quot;: 2048</span><br><span class="line">    &#125;,</span><br><span class="line">    &quot;names&quot;: [</span><br><span class="line">        &#123;</span><br><span class="line">            &quot;C&quot;: &quot;CN&quot;,</span><br><span class="line">            &quot;L&quot;: &quot;Beijing&quot;,</span><br><span class="line">            &quot;ST&quot;: &quot;Beijing&quot;</span><br><span class="line">        &#125;</span><br><span class="line">    ]</span><br><span class="line">&#125;</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>
<p>生成证书：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">[root@k8s-master1 etcd]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -</span><br><span class="line">2021/12/12 04:18:51 [INFO] generating a new CA key and certificate from CSR</span><br><span class="line">2021/12/12 04:18:51 [INFO] generate received request</span><br><span class="line">2021/12/12 04:18:51 [INFO] received CSR</span><br><span class="line">2021/12/12 04:18:51 [INFO] generating key: rsa-2048</span><br><span class="line">2021/12/12 04:18:51 [INFO] encoded CSR</span><br><span class="line">2021/12/12 04:18:51 [INFO] signed certificate with serial number 366484398928188099463200196143389891567502667552</span><br><span class="line">[root@k8s-master1 etcd]# ll</span><br><span class="line">total 20</span><br><span class="line">-rw-r--r--. 1 root root  287 Dec 12 04:01 ca-config.json</span><br><span class="line">-rw-r--r--. 1 root root  956 Dec 12 04:18 ca.csr</span><br><span class="line">-rw-r--r--. 1 root root  209 Dec 12 04:02 ca-csr.json</span><br><span class="line">-rw-------. 1 root root 1675 Dec 12 04:18 ca-key.pem</span><br><span class="line">-rw-r--r--. 1 root root 1265 Dec 12 04:18 ca.pem</span><br><span class="line"></span><br></pre></td></tr></table></figure>
<p><img src="https://img-blog.csdnimg.cn/14cc9d03f0e34d258d2242356b2bb440.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5rKn5rW35LmL5beF,size_20,color_FFFFFF,t_70,g_se,x_16" alt="在这里插入图片描述"><br>会生成ca.pem和ca-key.pem文件。</p>
<h3 id="2-使用自签CA签发Etcd-HTTPS证书"><a href="#2-使用自签CA签发Etcd-HTTPS证书" class="headerlink" title="2. 使用自签CA签发Etcd HTTPS证书"></a>2. 使用自签CA签发Etcd HTTPS证书</h3><p>创建证书申请文件：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; server-csr.json &lt;&lt; EOF</span><br><span class="line">&#123;</span><br><span class="line">    &quot;CN&quot;: &quot;etcd&quot;,</span><br><span class="line">    &quot;hosts&quot;: [</span><br><span class="line">    &quot;192.168.100.61&quot;,</span><br><span class="line">    &quot;192.168.100.62&quot;,</span><br><span class="line">    &quot;192.168.100.63&quot;,</span><br><span class="line">    &quot;192.168.100.64&quot;,</span><br><span class="line">    &quot;192.168.100.65&quot;,</span><br><span class="line">    &quot;192.168.100.66&quot;,</span><br><span class="line">    &quot;192.168.100.67&quot;</span><br><span class="line">    ],</span><br><span class="line">    &quot;key&quot;: &#123;</span><br><span class="line">        &quot;algo&quot;: &quot;rsa&quot;,</span><br><span class="line">        &quot;size&quot;: 2048</span><br><span class="line">    &#125;,</span><br><span class="line">    &quot;names&quot;: [</span><br><span class="line">        &#123;</span><br><span class="line">            &quot;C&quot;: &quot;CN&quot;,</span><br><span class="line">            &quot;L&quot;: &quot;BeiJing&quot;,</span><br><span class="line">            &quot;ST&quot;: &quot;BeiJing&quot;</span><br><span class="line">        &#125;</span><br><span class="line">    ]</span><br><span class="line">&#125;</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>
<p>注：上述文件hosts字段中IP为所有etcd节点的集群内部通信IP，一个都不能少！为了方便后期扩容可以多写几个预留的IP。</p>
<p>生成证书：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server</span><br></pre></td></tr></table></figure>
<p>会生成server.pem和server-key.pem文件。<br><img src="https://img-blog.csdnimg.cn/059149de6cf346c6acedc8765095d34f.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5rKn5rW35LmL5beF,size_20,color_FFFFFF,t_70,g_se,x_16" alt="在这里插入图片描述"></p>
<h2 id="2-3-从Github下载二进制文件"><a href="#2-3-从Github下载二进制文件" class="headerlink" title="2.3 从Github下载二进制文件"></a>2.3 从Github下载二进制文件</h2><p>下载地址：<a target="_blank" rel="noopener" href="https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz">https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz</a></p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line">[root@k8s-master1 ~]# wget https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz</span><br><span class="line">--2021-12-12 04:25:12--  https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz</span><br><span class="line">Resolving github.com (github.com)... 20.205.243.166</span><br><span class="line">Connecting to github.com (github.com)|20.205.243.166|:443... connected.</span><br><span class="line">HTTP request sent, awaiting response... 302 Found</span><br><span class="line">Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/11225014/7548e300-9b63-11ea-958d-569643dcaaea?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211212%2Fus-east-1%2Fs3%2Faws4_request&amp;X-Amz-Date=20211212T122524Z&amp;X-Amz-Expires=300&amp;X-Amz-Signature=b76af2df7138f3044f8f9ca1ecace49a034b1c59fed54d4f4fe3a50cd1b3c470&amp;X-Amz-SignedHeaders=host&amp;actor_id=0&amp;key_id=0&amp;repo_id=11225014&amp;response-content-disposition=attachment%3B%20filename%3Detcd-v3.4.9-linux-amd64.tar.gz&amp;response-content-type=application%2Foctet-stream [following]</span><br><span class="line">--2021-12-12 04:25:24--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/11225014/7548e300-9b63-11ea-958d-569643dcaaea?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211212%2Fus-east-1%2Fs3%2Faws4_request&amp;X-Amz-Date=20211212T122524Z&amp;X-Amz-Expires=300&amp;X-Amz-Signature=b76af2df7138f3044f8f9ca1ecace49a034b1c59fed54d4f4fe3a50cd1b3c470&amp;X-Amz-SignedHeaders=host&amp;actor_id=0&amp;key_id=0&amp;repo_id=11225014&amp;response-content-disposition=attachment%3B%20filename%3Detcd-v3.4.9-linux-amd64.tar.gz&amp;response-content-type=application%2Foctet-stream</span><br><span class="line">Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.108.133, ...</span><br><span class="line">Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected.</span><br><span class="line">HTTP request sent, awaiting response... 200 OK</span><br><span class="line">Length: 17364053 (17M) [application/octet-stream]</span><br><span class="line">Saving to: ‘etcd-v3.4.9-linux-amd64.tar.gz’</span><br><span class="line"></span><br><span class="line">100%[==============================================================================================================================================&gt;] 17,364,053   132KB/s   in 3m 42s</span><br><span class="line"></span><br><span class="line">2021-12-12 04:29:08 (76.3 KB/s) - ‘etcd-v3.4.9-linux-amd64.tar.gz’ saved [17364053/17364053]</span><br><span class="line"></span><br><span class="line">[root@k8s-master1 ~]#</span><br><span class="line">[root@k8s-master1 ~]# scp etcd-v3.4.9-linux-amd64.tar.gz root@192.168.100.62:/root</span><br><span class="line">root@192.168.100.62&#x27;s password:</span><br><span class="line">etcd-v3.4.9-linux-amd64.tar.gz    100%   17MB  65.9MB/s   00:00</span><br><span class="line">[root@k8s-master1 ~]# scp etcd-v3.4.9-linux-amd64.tar.gz root@192.168.100.63:/root</span><br><span class="line">root@192.168.100.63&#x27;s password:</span><br><span class="line">etcd-v3.4.9-linux-amd64.tar.gz    100%   17MB  65.4MB/s   00:00</span><br><span class="line">[root@k8s-master1 ~]#</span><br><span class="line"></span><br></pre></td></tr></table></figure>
<p><img src="https://img-blog.csdnimg.cn/f9b2ea515f694375a77939575b497f05.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5rKn5rW35LmL5beF,size_20,color_FFFFFF,t_70,g_se,x_16" alt="在这里插入图片描述"></p>
<h2 id="2-4-部署Etcd集群"><a href="#2-4-部署Etcd集群" class="headerlink" title="2.4 部署Etcd集群"></a>2.4 部署Etcd集群</h2><p>以下在节点1上操作，为简化操作，待会将节点1生成的所有文件拷贝到节点2和节点3.</p>
<h3 id="1-创建工作目录并解压二进制包"><a href="#1-创建工作目录并解压二进制包" class="headerlink" title="1. 创建工作目录并解压二进制包"></a>1. 创建工作目录并解压二进制包</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">mkdir /opt/etcd/&#123;bin,cfg,ssl&#125; -p</span><br><span class="line">tar zxvf etcd-v3.4.9-linux-amd64.tar.gz</span><br><span class="line">mv etcd-v3.4.9-linux-amd64/&#123;etcd,etcdctl&#125; /opt/etcd/bin/</span><br></pre></td></tr></table></figure>
<h3 id="2-创建etcd配置文件"><a href="#2-创建etcd配置文件" class="headerlink" title="2. 创建etcd配置文件"></a>2. 创建etcd配置文件</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br></pre></td><td class="code"><pre><span class="line"># 61 服务器</span><br><span class="line">cat &gt; /opt/etcd/cfg/etcd.conf &lt;&lt; EOF</span><br><span class="line">#[Member]</span><br><span class="line">ETCD_NAME=&quot;etcd-1&quot;</span><br><span class="line">ETCD_DATA_DIR=&quot;/var/lib/etcd/default.etcd&quot;</span><br><span class="line">ETCD_LISTEN_PEER_URLS=&quot;https://192.168.100.61:2380&quot;</span><br><span class="line">ETCD_LISTEN_CLIENT_URLS=&quot;https://192.168.100.61:2379&quot;</span><br><span class="line"></span><br><span class="line">#[Clustering]</span><br><span class="line">ETCD_INITIAL_ADVERTISE_PEER_URLS=&quot;https://192.168.100.61:2380&quot;</span><br><span class="line">ETCD_ADVERTISE_CLIENT_URLS=&quot;https://192.168.100.61:2379&quot;</span><br><span class="line">ETCD_INITIAL_CLUSTER=&quot;etcd-1=https://192.168.100.61:2380,etcd-2=https://192.168.100.62:2380,etcd-3=https://192.168.100.63:2380&quot;</span><br><span class="line">ETCD_INITIAL_CLUSTER_TOKEN=&quot;etcd-cluster&quot;</span><br><span class="line">ETCD_INITIAL_CLUSTER_STATE=&quot;new&quot;</span><br><span class="line">EOF</span><br><span class="line"></span><br><span class="line">#62 服务器</span><br><span class="line">cat &gt; /opt/etcd/cfg/etcd.conf &lt;&lt; EOF</span><br><span class="line">#[Member]</span><br><span class="line">ETCD_NAME=&quot;etcd-2&quot;</span><br><span class="line">ETCD_DATA_DIR=&quot;/var/lib/etcd/default.etcd&quot;</span><br><span class="line">ETCD_LISTEN_PEER_URLS=&quot;https://192.168.100.62:2380&quot;</span><br><span class="line">ETCD_LISTEN_CLIENT_URLS=&quot;https://192.168.100.62:2379&quot;</span><br><span class="line"></span><br><span class="line">#[Clustering]</span><br><span class="line">ETCD_INITIAL_ADVERTISE_PEER_URLS=&quot;https://192.168.100.62:2380&quot;</span><br><span class="line">ETCD_ADVERTISE_CLIENT_URLS=&quot;https://192.168.100.62:2379&quot;</span><br><span class="line">ETCD_INITIAL_CLUSTER=&quot;etcd-1=https://192.168.100.61:2380,etcd-2=https://192.168.100.62:2380,etcd-3=https://192.168.100.63:2380&quot;</span><br><span class="line">ETCD_INITIAL_CLUSTER_TOKEN=&quot;etcd-cluster&quot;</span><br><span class="line">ETCD_INITIAL_CLUSTER_STATE=&quot;new&quot;</span><br><span class="line">EOF</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">#63 服务器</span><br><span class="line">cat &gt; /opt/etcd/cfg/etcd.conf &lt;&lt; EOF</span><br><span class="line">#[Member]</span><br><span class="line">ETCD_NAME=&quot;etcd-3&quot;</span><br><span class="line">ETCD_DATA_DIR=&quot;/var/lib/etcd/default.etcd&quot;</span><br><span class="line">ETCD_LISTEN_PEER_URLS=&quot;https://192.168.100.63:2380&quot;</span><br><span class="line">ETCD_LISTEN_CLIENT_URLS=&quot;https://192.168.100.63:2379&quot;</span><br><span class="line"></span><br><span class="line">#[Clustering]</span><br><span class="line">ETCD_INITIAL_ADVERTISE_PEER_URLS=&quot;https://192.168.100.63:2380&quot;</span><br><span class="line">ETCD_ADVERTISE_CLIENT_URLS=&quot;https://192.168.100.63:2379&quot;</span><br><span class="line">ETCD_INITIAL_CLUSTER=&quot;etcd-1=https://192.168.100.61:2380,etcd-2=https://192.168.100.62:2380,etcd-3=https://192.168.100.63:2380&quot;</span><br><span class="line">ETCD_INITIAL_CLUSTER_TOKEN=&quot;etcd-cluster&quot;</span><br><span class="line">ETCD_INITIAL_CLUSTER_STATE=&quot;new&quot;</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>
<blockquote>
<p>ETCD_NAME：节点名称，集群中唯一<br>ETCD_DATA_DIR：数据目录<br>ETCD_LISTEN_PEER_URLS：集群通信监听地址<br>ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址<br>ETCD_INITIAL_ADVERTISE_PEERURLS：集群通告地址<br>ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址<br>ETCD_INITIAL_CLUSTER：集群节点地址<br>ETCD_INITIALCLUSTER_TOKEN：集群Token<br>ETCD_INITIALCLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群  </p>
</blockquote>
<h3 id="3-systemd管理etcd"><a href="#3-systemd管理etcd" class="headerlink" title="3. systemd管理etcd"></a>3. systemd管理etcd</h3><p>==原来的代码，有问题==</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; /usr/lib/systemd/system/etcd.service &lt;&lt; EOF</span><br><span class="line">[Unit]</span><br><span class="line">Description=Etcd Server</span><br><span class="line">After=network.target</span><br><span class="line">After=network-online.target</span><br><span class="line">Wants=network-online.target</span><br><span class="line"></span><br><span class="line">[Service]</span><br><span class="line">Type=notify</span><br><span class="line">EnvironmentFile=/opt/etcd/cfg/etcd.conf</span><br><span class="line">ExecStart=/opt/etcd/bin/etcd \</span><br><span class="line">--cert-file=/opt/etcd/ssl/server.pem \</span><br><span class="line">--key-file=/opt/etcd/ssl/server-key.pem \</span><br><span class="line">--peer-cert-file=/opt/etcd/ssl/server.pem \</span><br><span class="line">--peer-key-file=/opt/etcd/ssl/server-key.pem \</span><br><span class="line">--trusted-ca-file=/opt/etcd/ssl/ca.pem \</span><br><span class="line">--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \</span><br><span class="line">--logger=zap</span><br><span class="line">Restart=on-failure</span><br><span class="line">LimitNOFILE=65536</span><br><span class="line"></span><br><span class="line">[Install]</span><br><span class="line">WantedBy=multi-user.target</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>

<h3 id="4-拷贝刚才生成的证书"><a href="#4-拷贝刚才生成的证书" class="headerlink" title="4. 拷贝刚才生成的证书"></a>4. 拷贝刚才生成的证书</h3><p>把刚才生成的证书拷贝到配置文件中的路径：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">[root@k8s-node2 cfg]# cp ~/TLS/etcd/ca*pem ~/TLS/etcd/server*pem /opt/etcd/ssl/</span><br><span class="line">[root@k8s-node2 cfg]# ll /opt/etcd/ssl/</span><br><span class="line">total 16</span><br><span class="line">-rw-------. 1 root root 1675 Dec 12 04:40 ca-key.pem</span><br><span class="line">-rw-r--r--. 1 root root 1265 Dec 12 04:40 ca.pem</span><br><span class="line">-rw-------. 1 root root 1675 Dec 12 04:40 server-key.pem</span><br><span class="line">-rw-r--r--. 1 root root 1363 Dec 12 04:40 server.pem</span><br><span class="line"></span><br></pre></td></tr></table></figure>
<p><img src="https://img-blog.csdnimg.cn/6503b219f8f047bab07bb760400fc197.png" alt="在这里插入图片描述"></p>
<h3 id="5-启动并设置开机启动"><a href="#5-启动并设置开机启动" class="headerlink" title="5. 启动并设置开机启动"></a>5. 启动并设置开机启动</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">systemctl daemon-reload</span><br><span class="line">systemctl start etcd &amp;&amp; systemctl enable etcd</span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h3 id="6-将上面节点1所有生成的文件拷贝到节点2和节点3"><a href="#6-将上面节点1所有生成的文件拷贝到节点2和节点3" class="headerlink" title="6. 将上面节点1所有生成的文件拷贝到节点2和节点3"></a>6. 将上面节点1所有生成的文件拷贝到节点2和节点3</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">scp -r /opt/etcd/ root@192.168.100.62:/opt/</span><br><span class="line">scp /usr/lib/systemd/system/etcd.service root@192.168.100.62:/usr/lib/systemd/system/</span><br><span class="line">scp -r /opt/etcd/ root@192.168.100.63:/opt/</span><br><span class="line">scp /usr/lib/systemd/system/etcd.service root@192.168.100.63:/usr/lib/systemd/system/</span><br></pre></td></tr></table></figure>
<p>然后在节点2和节点3分别修改etcd.conf配置文件中的节点名称和当前服务器IP：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">vi /opt/etcd/cfg/etcd.conf</span><br><span class="line">#[Member]</span><br><span class="line">ETCD_NAME=&quot;etcd-1&quot;   # 修改此处，节点2改为etcd-2，节点3改为etcd-3</span><br><span class="line">ETCD_DATA_DIR=&quot;/var/lib/etcd/default.etcd&quot;</span><br><span class="line">ETCD_LISTEN_PEER_URLS=&quot;https://192.168.100.61:2380&quot;   # 修改此处为当前服务器IP</span><br><span class="line">ETCD_LISTEN_CLIENT_URLS=&quot;https://192.168.100.61:2379&quot; # 修改此处为当前服务器IP</span><br><span class="line"></span><br><span class="line">#[Clustering]</span><br><span class="line">ETCD_INITIAL_ADVERTISE_PEER_URLS=&quot;https://192.168.100.61:2380&quot; # 修改此处为当前服务器IP</span><br><span class="line">ETCD_ADVERTISE_CLIENT_URLS=&quot;https://192.168.100.61:2379&quot; # 修改此处为当前服务器IP</span><br><span class="line">ETCD_INITIAL_CLUSTER=&quot;etcd-1=https://192.168.100.61:2380,etcd-2=https://192.168.100.62:2380,etcd-3=https://192.168.100.63:2380&quot;</span><br><span class="line">ETCD_INITIAL_CLUSTER_TOKEN=&quot;etcd-cluster&quot;</span><br><span class="line">ETCD_INITIAL_CLUSTER_STATE=&quot;new&quot;</span><br></pre></td></tr></table></figure>
<p>最后启动etcd并设置开机启动，同上。</p>
<h3 id="7-查看集群状态"><a href="#7-查看集群状态" class="headerlink" title="7. 查看集群状态"></a>7. 查看集群状态</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">/opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints=&quot;https://192.168.100.61:2379,https://192.168.100.62:2379,https://192.168.100.63:2379&quot; endpoint health --write-out=table</span><br><span class="line"></span><br><span class="line">+----------------------------+--------+-------------+-------+</span><br><span class="line">|          ENDPOINT    | HEALTH |    TOOK     | ERROR |</span><br><span class="line">+----------------------------+--------+-------------+-------+</span><br><span class="line">| https://192.168.100.61:2379 |   true | 10.301506ms |    |</span><br><span class="line">| https://192.168.100.63:2379 |   true | 12.87467ms |     |</span><br><span class="line">| https://192.168.100.62:2379 |   true | 13.225954ms |    |</span><br><span class="line">+----------------------------+--------+-------------+-------+</span><br></pre></td></tr></table></figure>
<p>如果输出上面信息，就说明集群部署成功。<br>如果有问题第一步先看日志：/var/log/message 或 journalctl -u etcd</p>
<h1 id="三、安装Docker"><a href="#三、安装Docker" class="headerlink" title="三、安装Docker"></a>三、安装Docker</h1><p>这里使用Docker作为容器引擎，也可以换成别的，例如containerd<br>下载地址：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">wget https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz</span><br></pre></td></tr></table></figure>
<p>以下在所有节点操作。这里采用二进制安装，用yum安装也一样。</p>
<h2 id="3-1-解压二进制包"><a href="#3-1-解压二进制包" class="headerlink" title="3.1 解压二进制包"></a>3.1 解压二进制包</h2><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">tar zxvf docker-19.03.9.tgz</span><br><span class="line">mv docker/* /usr/bin</span><br></pre></td></tr></table></figure>
<h2 id="3-2-systemd管理docker"><a href="#3-2-systemd管理docker" class="headerlink" title="3.2 systemd管理docker"></a>3.2 systemd管理docker</h2><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; /usr/lib/systemd/system/docker.service &lt;&lt; EOF</span><br><span class="line">[Unit]</span><br><span class="line">Description=Docker Application Container Engine</span><br><span class="line">Documentation=https://docs.docker.com</span><br><span class="line">After=network-online.target firewalld.service</span><br><span class="line">Wants=network-online.target</span><br><span class="line"></span><br><span class="line">[Service]</span><br><span class="line">Type=notify</span><br><span class="line">ExecStart=/usr/bin/dockerd</span><br><span class="line">ExecReload=/bin/kill -s HUP $MAINPID</span><br><span class="line">LimitNOFILE=infinity</span><br><span class="line">LimitNPROC=infinity</span><br><span class="line">LimitCORE=infinity</span><br><span class="line">TimeoutStartSec=0</span><br><span class="line">Delegate=yes</span><br><span class="line">KillMode=process</span><br><span class="line">Restart=on-failure</span><br><span class="line">StartLimitBurst=3</span><br><span class="line">StartLimitInterval=60s</span><br><span class="line"></span><br><span class="line">[Install]</span><br><span class="line">WantedBy=multi-user.target</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>
<h2 id="3-3-创建配置文件"><a href="#3-3-创建配置文件" class="headerlink" title="3.3 创建配置文件"></a>3.3 创建配置文件</h2><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">mkdir /etc/docker</span><br><span class="line">cat &gt; /etc/docker/daemon.json &lt;&lt; EOF</span><br><span class="line">&#123;</span><br><span class="line">  &quot;registry-mirrors&quot;: [&quot;https://b9pmyelo.mirror.aliyuncs.com&quot;]</span><br><span class="line">&#125;</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>
<ul>
<li>registry-mirrors 阿里云镜像加速器</li>
<li><h2 id="3-4-启动并设置开机启动"><a href="#3-4-启动并设置开机启动" class="headerlink" title="3.4 启动并设置开机启动"></a>3.4 启动并设置开机启动</h2><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">systemctl daemon-reload</span><br><span class="line">systemctl start docker &amp;&amp; systemctl enable docker</span><br></pre></td></tr></table></figure>
<h1 id="四、部署Master-Node"><a href="#四、部署Master-Node" class="headerlink" title="四、部署Master Node"></a>四、部署Master Node</h1>如果你在学习中遇到问题或者文档有误可联系阿良~ 微信: k8init<h2 id="4-1-生成kube-apiserver证书"><a href="#4-1-生成kube-apiserver证书" class="headerlink" title="4.1 生成kube-apiserver证书"></a>4.1 生成kube-apiserver证书</h2><h3 id="1-自签证书颁发机构（CA）-1"><a href="#1-自签证书颁发机构（CA）-1" class="headerlink" title="1. 自签证书颁发机构（CA）"></a>1. 自签证书颁发机构（CA）</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br></pre></td><td class="code"><pre><span class="line">cd ~/TLS/k8s</span><br><span class="line"></span><br><span class="line">cat &gt; ca-config.json &lt;&lt; EOF</span><br><span class="line">&#123;</span><br><span class="line">  &quot;signing&quot;: &#123;</span><br><span class="line">    &quot;default&quot;: &#123;</span><br><span class="line">      &quot;expiry&quot;: &quot;87600h&quot;</span><br><span class="line">    &#125;,</span><br><span class="line">    &quot;profiles&quot;: &#123;</span><br><span class="line">      &quot;kubernetes&quot;: &#123;</span><br><span class="line">         &quot;expiry&quot;: &quot;87600h&quot;,</span><br><span class="line">         &quot;usages&quot;: [</span><br><span class="line">            &quot;signing&quot;,</span><br><span class="line">            &quot;key encipherment&quot;,</span><br><span class="line">            &quot;server auth&quot;,</span><br><span class="line">            &quot;client auth&quot;</span><br><span class="line">        ]</span><br><span class="line">      &#125;</span><br><span class="line">    &#125;</span><br><span class="line">  &#125;</span><br><span class="line">&#125;</span><br><span class="line">EOF</span><br><span class="line">cat &gt; ca-csr.json &lt;&lt; EOF</span><br><span class="line">&#123;</span><br><span class="line">    &quot;CN&quot;: &quot;kubernetes&quot;,</span><br><span class="line">    &quot;key&quot;: &#123;</span><br><span class="line">        &quot;algo&quot;: &quot;rsa&quot;,</span><br><span class="line">        &quot;size&quot;: 2048</span><br><span class="line">    &#125;,</span><br><span class="line">    &quot;names&quot;: [</span><br><span class="line">        &#123;</span><br><span class="line">            &quot;C&quot;: &quot;CN&quot;,</span><br><span class="line">            &quot;L&quot;: &quot;Beijing&quot;,</span><br><span class="line">            &quot;ST&quot;: &quot;Beijing&quot;,</span><br><span class="line">            &quot;O&quot;: &quot;k8s&quot;,</span><br><span class="line">            &quot;OU&quot;: &quot;System&quot;</span><br><span class="line">        &#125;</span><br><span class="line">    ]</span><br><span class="line">&#125;</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>
生成证书：<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">cfssl gencert -initca ca-csr.json | cfssljson -bare ca -</span><br></pre></td></tr></table></figure>
会生成ca.pem和ca-key.pem文件。</li>
</ul>
<h3 id="2-使用自签CA签发kube-apiserver-HTTPS证书"><a href="#2-使用自签CA签发kube-apiserver-HTTPS证书" class="headerlink" title="2. 使用自签CA签发kube-apiserver HTTPS证书"></a>2. 使用自签CA签发kube-apiserver HTTPS证书</h3><p>创建证书申请文件：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; server-csr.json &lt;&lt; EOF</span><br><span class="line">&#123;</span><br><span class="line">    &quot;CN&quot;: &quot;kubernetes&quot;,</span><br><span class="line">    &quot;hosts&quot;: [</span><br><span class="line">      &quot;10.0.0.1&quot;,</span><br><span class="line">      &quot;127.0.0.1&quot;,</span><br><span class="line">      &quot;192.168.100.61&quot;,</span><br><span class="line">      &quot;192.168.100.62&quot;,</span><br><span class="line">      &quot;192.168.100.63&quot;,</span><br><span class="line">      &quot;192.168.100.64&quot;,</span><br><span class="line">      &quot;192.168.100.65&quot;,</span><br><span class="line">      &quot;192.168.100.66&quot;,</span><br><span class="line">      &quot;192.168.100.67&quot;,</span><br><span class="line">      &quot;kubernetes&quot;,</span><br><span class="line">      &quot;kubernetes.default&quot;,</span><br><span class="line">      &quot;kubernetes.default.svc&quot;,</span><br><span class="line">      &quot;kubernetes.default.svc.cluster&quot;,</span><br><span class="line">      &quot;kubernetes.default.svc.cluster.local&quot;</span><br><span class="line">    ],</span><br><span class="line">    &quot;key&quot;: &#123;</span><br><span class="line">        &quot;algo&quot;: &quot;rsa&quot;,</span><br><span class="line">        &quot;size&quot;: 2048</span><br><span class="line">    &#125;,</span><br><span class="line">    &quot;names&quot;: [</span><br><span class="line">        &#123;</span><br><span class="line">            &quot;C&quot;: &quot;CN&quot;,</span><br><span class="line">            &quot;L&quot;: &quot;BeiJing&quot;,</span><br><span class="line">            &quot;ST&quot;: &quot;BeiJing&quot;,</span><br><span class="line">            &quot;O&quot;: &quot;k8s&quot;,</span><br><span class="line">            &quot;OU&quot;: &quot;System&quot;</span><br><span class="line">        &#125;</span><br><span class="line">    ]</span><br><span class="line">&#125;</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>
<p>注：上述文件hosts字段中IP为所有Master/LB/VIP IP，一个都不能少！为了方便后期扩容可以多写几个预留的IP。<br>生成证书：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server</span><br></pre></td></tr></table></figure>
<p>会生成server.pem和server-key.pem文件。</p>
<h2 id="4-2-从Github下载二进制文件"><a href="#4-2-从Github下载二进制文件" class="headerlink" title="4.2 从Github下载二进制文件"></a>4.2 从Github下载二进制文件</h2><p>下载地址： <a target="_blank" rel="noopener" href="https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md">https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md</a><br><img src="https://img-blog.csdnimg.cn/153632e27de145f2a9ada02dd176c5ca.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5rKn5rW35LmL5beF,size_20,color_FFFFFF,t_70,g_se,x_16" alt="在这里插入图片描述"><br><img src="https://img-blog.csdnimg.cn/e0423d5347ea4d969b32a2154d90adde.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5rKn5rW35LmL5beF,size_20,color_FFFFFF,t_70,g_se,x_16" alt="在这里插入图片描述"></p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">wget https://dl.k8s.io/v1.20.13/kubernetes-server-linux-amd64.tar.gz</span><br></pre></td></tr></table></figure>
<p>注：打开链接你会发现里面有很多包，下载一个server包就够了，包含了Master和Worker Node二进制文件。</p>
<h2 id="4-3-解压二进制包"><a href="#4-3-解压二进制包" class="headerlink" title="4.3 解压二进制包"></a>4.3 解压二进制包</h2><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">mkdir -p /opt/kubernetes/&#123;bin,cfg,ssl,logs&#125; </span><br><span class="line">tar zxvf kubernetes-server-linux-amd64.tar.gz</span><br><span class="line">cd kubernetes/server/bin</span><br><span class="line">cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin</span><br><span class="line">cp kubectl /usr/bin/</span><br></pre></td></tr></table></figure>
<h2 id="4-4-部署kube-apiserver-master"><a href="#4-4-部署kube-apiserver-master" class="headerlink" title="4.4 部署kube-apiserver(master)"></a>4.4 部署kube-apiserver(master)</h2><h3 id="1-创建配置文件"><a href="#1-创建配置文件" class="headerlink" title="1. 创建配置文件"></a>1. 创建配置文件</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; /opt/kubernetes/cfg/kube-apiserver.conf &lt;&lt; EOF</span><br><span class="line">KUBE_APISERVER_OPTS=&quot;--logtostderr=false \\</span><br><span class="line">--v=2 \\</span><br><span class="line">--log-dir=/opt/kubernetes/logs \\</span><br><span class="line">--etcd-servers=https://192.168.100.61:2379,https://192.168.100.62:2379,https://192.168.100.63:2379 \\</span><br><span class="line">--bind-address=192.168.100.61 \\</span><br><span class="line">--secure-port=6443 \\</span><br><span class="line">--advertise-address=192.168.100.61 \\</span><br><span class="line">--allow-privileged=true \\</span><br><span class="line">--service-cluster-ip-range=10.0.0.0/24 \\</span><br><span class="line">--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\</span><br><span class="line">--authorization-mode=RBAC,Node \\</span><br><span class="line">--enable-bootstrap-token-auth=true \\</span><br><span class="line">--token-auth-file=/opt/kubernetes/cfg/token.csv \\</span><br><span class="line">--service-node-port-range=30000-32767 \\</span><br><span class="line">--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \\</span><br><span class="line">--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \\</span><br><span class="line">--tls-cert-file=/opt/kubernetes/ssl/server.pem  \\</span><br><span class="line">--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\</span><br><span class="line">--client-ca-file=/opt/kubernetes/ssl/ca.pem \\</span><br><span class="line">--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\</span><br><span class="line">--service-account-issuer=api \\</span><br><span class="line">--service-account-signing-key-file=/opt/kubernetes/ssl/server-key.pem \\</span><br><span class="line">--etcd-cafile=/opt/etcd/ssl/ca.pem \\</span><br><span class="line">--etcd-certfile=/opt/etcd/ssl/server.pem \\</span><br><span class="line">--etcd-keyfile=/opt/etcd/ssl/server-key.pem \\</span><br><span class="line">--requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \\</span><br><span class="line">--proxy-client-cert-file=/opt/kubernetes/ssl/server.pem \\</span><br><span class="line">--proxy-client-key-file=/opt/kubernetes/ssl/server-key.pem \\</span><br><span class="line">--requestheader-allowed-names=kubernetes \\</span><br><span class="line">--requestheader-extra-headers-prefix=X-Remote-Extra- \\</span><br><span class="line">--requestheader-group-headers=X-Remote-Group \\</span><br><span class="line">--requestheader-username-headers=X-Remote-User \\</span><br><span class="line">--enable-aggregator-routing=true \\</span><br><span class="line">--audit-log-maxage=30 \\</span><br><span class="line">--audit-log-maxbackup=3 \\</span><br><span class="line">--audit-log-maxsize=100 \\</span><br><span class="line">--audit-log-path=/opt/kubernetes/logs/k8s-audit.log&quot;</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>
<p>注：上面两个\ \ 第一个是转义符，第二个是换行符，使用转义符是为了使用EOF保留换行符。</p>
<blockquote>
<p>–logtostderr：启用日志<br>–v：日志等级<br>–log-dir：日志目录<br>–etcd-servers：etcd集群地址<br>–bind-address：监听地址<br>–secure-port：https安全端口<br>–advertise-address：集群通告地址<br>–allow-privileged：启用授权<br>–service-cluster-ip-range：Service虚拟IP地址段<br>–enable-admission-plugins：准入控制模块<br>–authorization-mode：认证授权，启用RBAC授权和节点自管理<br>–enable-bootstrap-token-auth：启用TLS bootstrap机制<br>–token-auth-file：bootstrap token文件<br>–service-node-port-range：Service nodeport类型默认分配端口范围<br>–kubelet-client-xxx：apiserver访问kubelet客户端证书<br>–tls-xxx-file：apiserver https证书<br>1.20版本必须加的参数：–service-account-issuer，–service-account-signing-key-file<br>–etcd-xxxfile：连接Etcd集群证书<br>–audit-log-xxx：审计日志<br>启动聚合层相关配置：–requestheader-client-ca-file，–proxy-client-cert-file，–proxy-client-key-file，–requestheader-allowed-names，–requestheader-extra-headers-prefix，–requestheader-group-headers，–requestheader-username-headers，–enable-aggregator-routing  </p>
</blockquote>
<h3 id="2-拷贝刚才生成的证书"><a href="#2-拷贝刚才生成的证书" class="headerlink" title="2. 拷贝刚才生成的证书"></a>2. 拷贝刚才生成的证书</h3><p>把刚才生成的证书拷贝到配置文件中的路径：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">cp ~/TLS/k8s/ca*pem ~/TLS/k8s/server*pem /opt/kubernetes/ssl/</span><br></pre></td></tr></table></figure>

<h3 id="3-启用-TLS-Bootstrapping-机制"><a href="#3-启用-TLS-Bootstrapping-机制" class="headerlink" title="3. 启用 TLS Bootstrapping 机制"></a>3. 启用 TLS Bootstrapping 机制</h3><p>TLS Bootstraping：Master apiserver启用TLS认证后，Node节点kubelet和kube-proxy要与kube-apiserver进行通信，必须使用CA签发的有效证书才可以，当Node节点很多时，这种客户端证书颁发需要大量工作，同样也会增加集群扩展复杂度。为了简化流程，Kubernetes引入了TLS bootstraping机制来自动颁发客户端证书，kubelet会以一个低权限用户自动向apiserver申请证书，kubelet的证书由apiserver动态签署。所以强烈建议在Node上使用这种方式，目前主要用于kubelet，kube-proxy还是由我们统一颁发一个证书。<br>TLS bootstraping 工作流程：<br><img src="https://img-blog.csdnimg.cn/77f398785c824a76900e4b6fdf2e5deb.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5rKn5rW35LmL5beF,size_19,color_FFFFFF,t_70,g_se,x_16" alt="在这里插入图片描述"></p>
<p>创建上述配置文件中token文件：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; /opt/kubernetes/cfg/token.csv &lt;&lt; EOF</span><br><span class="line">c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,&quot;system:node-bootstrapper&quot;</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>
<p>格式：token，用户名，UID，用户组<br>token也可自行生成替换：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">head -c 16 /dev/urandom | od -An -t x | tr -d &#x27; &#x27;</span><br></pre></td></tr></table></figure>
<h3 id="4-systemd管理apiserver"><a href="#4-systemd管理apiserver" class="headerlink" title="4. systemd管理apiserver"></a>4. systemd管理apiserver</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; /usr/lib/systemd/system/kube-apiserver.service &lt;&lt; EOF</span><br><span class="line">[Unit]</span><br><span class="line">Description=Kubernetes API Server</span><br><span class="line">Documentation=https://github.com/kubernetes/kubernetes</span><br><span class="line"></span><br><span class="line">[Service]</span><br><span class="line">EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf</span><br><span class="line">ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS</span><br><span class="line">Restart=on-failure</span><br><span class="line"></span><br><span class="line">[Install]</span><br><span class="line">WantedBy=multi-user.target</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>
<h3 id="5-启动并设置开机启动-1"><a href="#5-启动并设置开机启动-1" class="headerlink" title="5. 启动并设置开机启动"></a>5. 启动并设置开机启动</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">[root@k8s-master1 bin]# systemctl daemon-reload</span><br><span class="line">[root@k8s-master1 bin]# systemctl start kube-apiserver &amp;&amp; systemctl enable kube-apiserver</span><br><span class="line"></span><br><span class="line">[root@k8s-master1 bin]# ps -ef|grep kube-apiserver</span><br><span class="line">root      12328      1 52 16:53 ?        00:00:05 /opt/kubernetes/bin/kube-apiserver --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --etcd-servers=https://192.168.100.61:2379,https://192.168.100.62:2379,https://192.168.100.63:2379 --bind-address=192.168.100.61 --secure-port=6443 --advertise-address=192.168.100.61 --allow-privileged=true --service-cluster-ip-range=10.0.0.0/24 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction --authorization-mode=RBAC,Node --enable-bootstrap-token-auth=true --token-auth-file=/opt/kubernetes/cfg/token.csv --service-node-port-range=30000-32767 --kubelet-client-certificate=/opt/kubernetes/ssl/server.pem --kubelet-client-key=/opt/kubernetes/ssl/server-key.pem --tls-cert-file=/opt/kubernetes/ssl/server.pem --tls-private-key-file=/opt/kubernetes/ssl/server-key.pem --client-ca-file=/opt/kubernetes/ssl/ca.pem --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem --service-account-issuer=api --service-account-signing-key-file=/opt/kubernetes/ssl/server-key.pem --etcd-cafile=/opt/etcd/ssl/ca.pem --etcd-certfile=/opt/etcd/ssl/server.pem --etcd-keyfile=/opt/etcd/ssl/server-key.pem --requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem --proxy-client-cert-file=/opt/kubernetes/ssl/server.pem --proxy-client-key-file=/opt/kubernetes/ssl/server-key.pem --requestheader-allowed-names=kubernetes --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --enable-aggregator-routing=true --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/opt/kubernetes/logs/k8s-audit.log</span><br><span class="line">root      12368  11582  0 16:53 pts/1    00:00:00 grep --color=auto kube-apiserver</span><br><span class="line">[root@k8s-master1 bin]#</span><br><span class="line"></span><br></pre></td></tr></table></figure>

<h2 id="4-5-部署kube-controller-manager"><a href="#4-5-部署kube-controller-manager" class="headerlink" title="4.5 部署kube-controller-manager"></a>4.5 部署kube-controller-manager</h2><h3 id="1-创建配置文件-1"><a href="#1-创建配置文件-1" class="headerlink" title="1. 创建配置文件"></a>1. 创建配置文件</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; /opt/kubernetes/cfg/kube-controller-manager.conf &lt;&lt; EOF</span><br><span class="line">KUBE_CONTROLLER_MANAGER_OPTS=&quot;--logtostderr=false \\</span><br><span class="line">--v=2 \\</span><br><span class="line">--log-dir=/opt/kubernetes/logs \\</span><br><span class="line">--leader-elect=true \\</span><br><span class="line">--kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \\</span><br><span class="line">--bind-address=127.0.0.1 \\</span><br><span class="line">--allocate-node-cidrs=true \\</span><br><span class="line">--cluster-cidr=10.244.0.0/16 \\</span><br><span class="line">--service-cluster-ip-range=10.0.0.0/24 \\</span><br><span class="line">--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\</span><br><span class="line">--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem  \\</span><br><span class="line">--root-ca-file=/opt/kubernetes/ssl/ca.pem \\</span><br><span class="line">--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\</span><br><span class="line">--cluster-signing-duration=87600h0m0s&quot;</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>
<blockquote>
<p>–kubeconfig：连接apiserver配置文件<br>–leader-elect：当该组件启动多个时，自动选举（HA）<br>–cluster-signing-cert-file/–cluster-signing-key-file：自动为kubelet颁发证书的CA，与apiserver保持一致  </p>
</blockquote>
<h3 id="2-生成kubeconfig文件"><a href="#2-生成kubeconfig文件" class="headerlink" title="2. 生成kubeconfig文件"></a>2. 生成kubeconfig文件</h3><p>生成kube-controller-manager证书：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"># 切换工作目录</span><br><span class="line">cd ~/TLS/k8s</span><br><span class="line"></span><br><span class="line"># 创建证书请求文件</span><br><span class="line">cat &gt; kube-controller-manager-csr.json &lt;&lt; EOF</span><br><span class="line">&#123;</span><br><span class="line">  &quot;CN&quot;: &quot;system:kube-controller-manager&quot;,</span><br><span class="line">  &quot;hosts&quot;: [],</span><br><span class="line">  &quot;key&quot;: &#123;</span><br><span class="line">    &quot;algo&quot;: &quot;rsa&quot;,</span><br><span class="line">    &quot;size&quot;: 2048</span><br><span class="line">  &#125;,</span><br><span class="line">  &quot;names&quot;: [</span><br><span class="line">    &#123;</span><br><span class="line">      &quot;C&quot;: &quot;CN&quot;,</span><br><span class="line">      &quot;L&quot;: &quot;BeiJing&quot;, </span><br><span class="line">      &quot;ST&quot;: &quot;BeiJing&quot;,</span><br><span class="line">      &quot;O&quot;: &quot;system:masters&quot;,</span><br><span class="line">      &quot;OU&quot;: &quot;System&quot;</span><br><span class="line">    &#125;</span><br><span class="line">  ]</span><br><span class="line">&#125;</span><br><span class="line">EOF</span><br><span class="line"></span><br><span class="line"># 生成证书</span><br><span class="line">cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager</span><br></pre></td></tr></table></figure>
<p>生成kubeconfig文件（以下是shell命令，直接在终端执行）：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">KUBE_CONFIG=&quot;/opt/kubernetes/cfg/kube-controller-manager.kubeconfig&quot;</span><br><span class="line">KUBE_APISERVER=&quot;https://192.168.100.61:6443&quot;</span><br><span class="line"></span><br><span class="line">kubectl config set-cluster kubernetes \</span><br><span class="line">  --certificate-authority=/opt/kubernetes/ssl/ca.pem \</span><br><span class="line">  --embed-certs=true \</span><br><span class="line">  --server=$&#123;KUBE_APISERVER&#125; \</span><br><span class="line">  --kubeconfig=$&#123;KUBE_CONFIG&#125;</span><br><span class="line">kubectl config set-credentials kube-controller-manager \</span><br><span class="line">  --client-certificate=./kube-controller-manager.pem \</span><br><span class="line">  --client-key=./kube-controller-manager-key.pem \</span><br><span class="line">  --embed-certs=true \</span><br><span class="line">  --kubeconfig=$&#123;KUBE_CONFIG&#125;</span><br><span class="line">kubectl config set-context default \</span><br><span class="line">  --cluster=kubernetes \</span><br><span class="line">  --user=kube-controller-manager \</span><br><span class="line">  --kubeconfig=$&#123;KUBE_CONFIG&#125;</span><br><span class="line">kubectl config use-context default --kubeconfig=$&#123;KUBE_CONFIG&#125;</span><br></pre></td></tr></table></figure>

<p>执行效果demo：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br></pre></td><td class="code"><pre><span class="line">[root@k8s-master1 k8s]# KUBE_CONFIG=&quot;/opt/kubernetes/cfg/kube-controller-manager.kubeconfig&quot;</span><br><span class="line">[root@k8s-master1 k8s]# KUBE_APISERVER=&quot;https://192.168.100.61:6443&quot;</span><br><span class="line">[root@k8s-master1 k8s]#</span><br><span class="line">[root@k8s-master1 k8s]# kubectl config set-cluster kubernetes \</span><br><span class="line">&gt;   --certificate-authority=/opt/kubernetes/ssl/ca.pem \</span><br><span class="line">&gt;   --embed-certs=true \</span><br><span class="line">&gt;   --server=$&#123;KUBE_APISERVER&#125; \</span><br><span class="line">&gt;   --kubeconfig=$&#123;KUBE_CONFIG&#125;</span><br><span class="line">Cluster &quot;kubernetes&quot; set.</span><br><span class="line">[root@k8s-master1 k8s]# kubectl config set-credentials kube-controller-manager \</span><br><span class="line">&gt;   --client-certificate=./kube-controller-manager.pem \</span><br><span class="line">&gt;   --client-key=./kube-controller-manager-key.pem \</span><br><span class="line">&gt;   --embed-certs=true \</span><br><span class="line">&gt;   --kubeconfig=$&#123;KUBE_CONFIG&#125;</span><br><span class="line">User &quot;kube-controller-manager&quot; set.</span><br><span class="line">[root@k8s-master1 k8s]# kubectl config set-context default \</span><br><span class="line">&gt;   --cluster=kubernetes \</span><br><span class="line">&gt;   --user=kube-controller-manager \</span><br><span class="line">&gt;   --kubeconfig=$&#123;KUBE_CONFIG&#125;</span><br><span class="line">Context &quot;default&quot; created.</span><br><span class="line">[root@k8s-master1 k8s]# kubectl config use-context default --kubeconfig=$&#123;KUBE_CONFIG&#125;</span><br><span class="line">Switched to context &quot;default&quot;.</span><br><span class="line">[root@k8s-master1 k8s]# </span><br><span class="line">[root@k8s-master1 k8s]# ll /opt/kubernetes/cfg/</span><br><span class="line">total 20</span><br><span class="line">-rw-r--r--. 1 root root 1709 Dec 12 16:51 kube-apiserver.conf</span><br><span class="line">-rw-r--r--. 1 root root  582 Dec 12 16:55 kube-controller-manager.conf</span><br><span class="line">-rw-------. 1 root root 6344 Dec 12 16:56 kube-controller-manager.kubeconfig</span><br><span class="line">-rw-r--r--. 1 root root   84 Dec 12 16:52 token.csv</span><br><span class="line">[root@k8s-master1 k8s]#</span><br><span class="line"></span><br></pre></td></tr></table></figure>

<h3 id="3-systemd管理controller-manager"><a href="#3-systemd管理controller-manager" class="headerlink" title="3. systemd管理controller-manager"></a>3. systemd管理controller-manager</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; /usr/lib/systemd/system/kube-controller-manager.service &lt;&lt; EOF</span><br><span class="line">[Unit]</span><br><span class="line">Description=Kubernetes Controller Manager</span><br><span class="line">Documentation=https://github.com/kubernetes/kubernetes</span><br><span class="line"></span><br><span class="line">[Service]</span><br><span class="line">EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf</span><br><span class="line">ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS</span><br><span class="line">Restart=on-failure</span><br><span class="line"></span><br><span class="line">[Install]</span><br><span class="line">WantedBy=multi-user.target</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>
<h3 id="4-启动并设置开机启动"><a href="#4-启动并设置开机启动" class="headerlink" title="4. 启动并设置开机启动"></a>4. 启动并设置开机启动</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">[root@k8s-master1 k8s]# systemctl daemon-reload</span><br><span class="line">[root@k8s-master1 k8s]# systemctl start kube-controller-manager &amp;&amp; systemctl enable kube-controller-manager</span><br><span class="line">[root@k8s-master1 k8s]#</span><br><span class="line">[root@k8s-master1 k8s]# ps -ef|grep kube-controller-manager</span><br><span class="line">root      12601      1  1 17:00 ?        00:03:55 /opt/kubernetes/bin/kube-controller-manager --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --leader-elect=true --kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig --bind-address=127.0.0.1 --allocate-node-cidrs=true --cluster-cidr=10.244.0.0/16 --service-cluster-ip-range=10.0.0.0/24 --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem --root-ca-file=/opt/kubernetes/ssl/ca.pem --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem --cluster-signing-duration=87600h0m0s</span><br><span class="line">root      18343  13439  0 20:53 pts/1    00:00:00 grep --color=auto kube-controller-manager</span><br><span class="line">[root@k8s-master1 k8s]#tail -f </span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="4-6-部署kube-scheduler"><a href="#4-6-部署kube-scheduler" class="headerlink" title="4.6 部署kube-scheduler"></a>4.6 部署kube-scheduler</h2><h3 id="1-创建配置文件-2"><a href="#1-创建配置文件-2" class="headerlink" title="1. 创建配置文件"></a>1. 创建配置文件</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; /opt/kubernetes/cfg/kube-scheduler.conf &lt;&lt; EOF</span><br><span class="line">KUBE_SCHEDULER_OPTS=&quot;--logtostderr=false \\</span><br><span class="line">--v=2 \\</span><br><span class="line">--log-dir=/opt/kubernetes/logs \\</span><br><span class="line">--leader-elect \\</span><br><span class="line">--kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig \\</span><br><span class="line">--bind-address=127.0.0.1&quot;</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>
<blockquote>
<p>–kubeconfig：连接apiserver配置文件<br>–leader-elect：当该组件启动多个时，自动选举（HA）  </p>
</blockquote>
<h3 id="2-生成kubeconfig文件-1"><a href="#2-生成kubeconfig文件-1" class="headerlink" title="2. 生成kubeconfig文件"></a>2. 生成kubeconfig文件</h3><p>生成kube-scheduler证书：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"># 切换工作目录</span><br><span class="line">cd ~/TLS/k8s</span><br><span class="line"></span><br><span class="line"># 创建证书请求文件</span><br><span class="line">cat &gt; kube-scheduler-csr.json &lt;&lt; EOF</span><br><span class="line">&#123;</span><br><span class="line">  &quot;CN&quot;: &quot;system:kube-scheduler&quot;,</span><br><span class="line">  &quot;hosts&quot;: [],</span><br><span class="line">  &quot;key&quot;: &#123;</span><br><span class="line">    &quot;algo&quot;: &quot;rsa&quot;,</span><br><span class="line">    &quot;size&quot;: 2048</span><br><span class="line">  &#125;,</span><br><span class="line">  &quot;names&quot;: [</span><br><span class="line">    &#123;</span><br><span class="line">      &quot;C&quot;: &quot;CN&quot;,</span><br><span class="line">      &quot;L&quot;: &quot;BeiJing&quot;,</span><br><span class="line">      &quot;ST&quot;: &quot;BeiJing&quot;,</span><br><span class="line">      &quot;O&quot;: &quot;system:masters&quot;,</span><br><span class="line">      &quot;OU&quot;: &quot;System&quot;</span><br><span class="line">    &#125;</span><br><span class="line">  ]</span><br><span class="line">&#125;</span><br><span class="line">EOF</span><br><span class="line"></span><br><span class="line"># 生成证书</span><br><span class="line">cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler</span><br></pre></td></tr></table></figure>
<p>生成kubeconfig文件：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">KUBE_CONFIG=&quot;/opt/kubernetes/cfg/kube-scheduler.kubeconfig&quot;</span><br><span class="line">KUBE_APISERVER=&quot;https://192.168.100.61:6443&quot;</span><br><span class="line"></span><br><span class="line">kubectl config set-cluster kubernetes \</span><br><span class="line">  --certificate-authority=/opt/kubernetes/ssl/ca.pem \</span><br><span class="line">  --embed-certs=true \</span><br><span class="line">  --server=$&#123;KUBE_APISERVER&#125; \</span><br><span class="line">  --kubeconfig=$&#123;KUBE_CONFIG&#125;</span><br><span class="line">kubectl config set-credentials kube-scheduler \</span><br><span class="line">  --client-certificate=./kube-scheduler.pem \</span><br><span class="line">  --client-key=./kube-scheduler-key.pem \</span><br><span class="line">  --embed-certs=true \</span><br><span class="line">  --kubeconfig=$&#123;KUBE_CONFIG&#125;</span><br><span class="line">kubectl config set-context default \</span><br><span class="line">  --cluster=kubernetes \</span><br><span class="line">  --user=kube-scheduler \</span><br><span class="line">  --kubeconfig=$&#123;KUBE_CONFIG&#125;</span><br><span class="line">kubectl config use-context default --kubeconfig=$&#123;KUBE_CONFIG&#125;</span><br></pre></td></tr></table></figure>

<h3 id="3-systemd管理scheduler"><a href="#3-systemd管理scheduler" class="headerlink" title="3. systemd管理scheduler"></a>3. systemd管理scheduler</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; /usr/lib/systemd/system/kube-scheduler.service &lt;&lt; EOF</span><br><span class="line">[Unit]</span><br><span class="line">Description=Kubernetes Scheduler</span><br><span class="line">Documentation=https://github.com/kubernetes/kubernetes</span><br><span class="line"></span><br><span class="line">[Service]</span><br><span class="line">EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf</span><br><span class="line">ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS</span><br><span class="line">Restart=on-failure</span><br><span class="line"></span><br><span class="line">[Install]</span><br><span class="line">WantedBy=multi-user.target</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>
<h3 id="4-启动并设置开机启动-1"><a href="#4-启动并设置开机启动-1" class="headerlink" title="4. 启动并设置开机启动"></a>4. 启动并设置开机启动</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">systemctl daemon-reload</span><br><span class="line">systemctl start kube-scheduler &amp;&amp; systemctl enable kube-scheduler</span><br></pre></td></tr></table></figure>
<h3 id="5-查看集群状态"><a href="#5-查看集群状态" class="headerlink" title="5. 查看集群状态"></a>5. 查看集群状态</h3><p>生成kubectl连接集群的证书：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; admin-csr.json &lt;&lt;EOF</span><br><span class="line">&#123;</span><br><span class="line">  &quot;CN&quot;: &quot;admin&quot;,</span><br><span class="line">  &quot;hosts&quot;: [],</span><br><span class="line">  &quot;key&quot;: &#123;</span><br><span class="line">    &quot;algo&quot;: &quot;rsa&quot;,</span><br><span class="line">    &quot;size&quot;: 2048</span><br><span class="line">  &#125;,</span><br><span class="line">  &quot;names&quot;: [</span><br><span class="line">    &#123;</span><br><span class="line">      &quot;C&quot;: &quot;CN&quot;,</span><br><span class="line">      &quot;L&quot;: &quot;BeiJing&quot;,</span><br><span class="line">      &quot;ST&quot;: &quot;BeiJing&quot;,</span><br><span class="line">      &quot;O&quot;: &quot;system:masters&quot;,</span><br><span class="line">      &quot;OU&quot;: &quot;System&quot;</span><br><span class="line">    &#125;</span><br><span class="line">  ]</span><br><span class="line">&#125;</span><br><span class="line">EOF</span><br><span class="line"></span><br><span class="line">cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin</span><br></pre></td></tr></table></figure>
<p>生成kubeconfig文件：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line">mkdir /root/.kube</span><br><span class="line"></span><br><span class="line">KUBE_CONFIG=&quot;/root/.kube/config&quot;</span><br><span class="line">KUBE_APISERVER=&quot;https://192.168.100.61:6443&quot;</span><br><span class="line"></span><br><span class="line">kubectl config set-cluster kubernetes \</span><br><span class="line">  --certificate-authority=/opt/kubernetes/ssl/ca.pem \</span><br><span class="line">  --embed-certs=true \</span><br><span class="line">  --server=$&#123;KUBE_APISERVER&#125; \</span><br><span class="line">  --kubeconfig=$&#123;KUBE_CONFIG&#125;</span><br><span class="line">kubectl config set-credentials cluster-admin \</span><br><span class="line">  --client-certificate=./admin.pem \</span><br><span class="line">  --client-key=./admin-key.pem \</span><br><span class="line">  --embed-certs=true \</span><br><span class="line">  --kubeconfig=$&#123;KUBE_CONFIG&#125;</span><br><span class="line">kubectl config set-context default \</span><br><span class="line">  --cluster=kubernetes \</span><br><span class="line">  --user=cluster-admin \</span><br><span class="line">  --kubeconfig=$&#123;KUBE_CONFIG&#125;</span><br><span class="line">kubectl config use-context default --kubeconfig=$&#123;KUBE_CONFIG&#125;</span><br></pre></td></tr></table></figure>
<p>通过kubectl工具查看当前集群组件状态：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">kubectl get cs</span><br><span class="line">NAME                STATUS    MESSAGE             ERROR</span><br><span class="line">scheduler             Healthy   ok                  </span><br><span class="line">controller-manager       Healthy   ok                  </span><br><span class="line">etcd-2               Healthy   &#123;&quot;health&quot;:&quot;true&quot;&#125;   </span><br><span class="line">etcd-1               Healthy   &#123;&quot;health&quot;:&quot;true&quot;&#125;   </span><br><span class="line">etcd-0               Healthy   &#123;&quot;health&quot;:&quot;true&quot;&#125;  </span><br></pre></td></tr></table></figure>
<p>如上输出说明Master节点组件运行正常。</p>
<h3 id="6-授权kubelet-bootstrap用户允许请求证书"><a href="#6-授权kubelet-bootstrap用户允许请求证书" class="headerlink" title="6. 授权kubelet-bootstrap用户允许请求证书"></a>6. 授权kubelet-bootstrap用户允许请求证书</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">kubectl create clusterrolebinding kubelet-bootstrap \</span><br><span class="line">--clusterrole=system:node-bootstrapper \</span><br><span class="line">--user=kubelet-bootstrap</span><br></pre></td></tr></table></figure>
<p>如果提示已存在，<a target="_blank" rel="noopener" href="https://blog.csdn.net/weixin_47153988/article/details/108866095">参考</a></p>
<h1 id="五、部署Worker-Node"><a href="#五、部署Worker-Node" class="headerlink" title="五、部署Worker Node"></a>五、部署Worker Node</h1><p>如果你在学习中遇到问题或者文档有误可联系阿良~ 微信: k8init<br>==下面还是在Master Node上操作，即同时作为Worker Node==</p>
<h2 id="5-1-创建工作目录并拷贝二进制文件"><a href="#5-1-创建工作目录并拷贝二进制文件" class="headerlink" title="5.1 创建工作目录并拷贝二进制文件"></a>5.1 创建工作目录并拷贝二进制文件</h2><p>在所有worker node创建工作目录：【Node】</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">mkdir -p /opt/kubernetes/&#123;bin,cfg,ssl,logs&#125; </span><br><span class="line">mkdir -p kubernetes/server/bin</span><br></pre></td></tr></table></figure>
<p>从master节点拷贝：【Master】</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">cd ~/kubernetes/server/bin</span><br><span class="line">scp -r * root@192.168.100.62:~/kubernetes/server/bin/ #需要提前在Node节点创建好目录</span><br><span class="line">scp -r * root@192.168.100.63:~/kubernetes/server/bin/</span><br></pre></td></tr></table></figure>
<p><img src="https://img-blog.csdnimg.cn/9bd094323bb34b51a392b440bd8ee865.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5rKn5rW35LmL5beF,size_20,color_FFFFFF,t_70,g_se,x_16" alt="在这里插入图片描述"><br><img src="https://img-blog.csdnimg.cn/ed4c2c7527a64b3796c016b300efc3a0.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5rKn5rW35LmL5beF,size_20,color_FFFFFF,t_70,g_se,x_16" alt="在这里插入图片描述"></p>
<p>Worker Node节点操作【Node】</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">cd ~/kubernetes/server/bin</span><br><span class="line">cp kubelet kube-proxy /opt/kubernetes/bin   # 需要提前将Master中的/opt/kubernetes/bin/*复制过来， 然NOde中本地拷贝</span><br></pre></td></tr></table></figure>
<h2 id="5-2-部署kubelet"><a href="#5-2-部署kubelet" class="headerlink" title="5.2 部署kubelet"></a>5.2 部署kubelet</h2><h3 id="1-创建配置文件-3"><a href="#1-创建配置文件-3" class="headerlink" title="1. 创建配置文件"></a>1. 创建配置文件</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; /opt/kubernetes/cfg/kubelet.conf &lt;&lt; EOF</span><br><span class="line">KUBELET_OPTS=&quot;--logtostderr=false \\</span><br><span class="line">--v=2 \\</span><br><span class="line">--log-dir=/opt/kubernetes/logs \\</span><br><span class="line">--hostname-override=k8s-master1 \\</span><br><span class="line">--network-plugin=cni \\</span><br><span class="line">--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\</span><br><span class="line">--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\</span><br><span class="line">--config=/opt/kubernetes/cfg/kubelet-config.yml \\</span><br><span class="line">--cert-dir=/opt/kubernetes/ssl \\</span><br><span class="line">--pod-infra-container-image=lizhenliang/pause-amd64:3.0&quot;</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>
<blockquote>
<p>–hostname-override：显示名称，集群中唯一<br>–network-plugin：启用CNI<br>–kubeconfig：空路径，会自动生成，后面用于连接apiserver<br>–bootstrap-kubeconfig：首次启动向apiserver申请证书<br>–config：配置参数文件<br>–cert-dir：kubelet证书生成目录<br>–pod-infra-container-image：管理Pod网络容器的镜像  </p>
</blockquote>
<h3 id="2-配置参数文件"><a href="#2-配置参数文件" class="headerlink" title="2. 配置参数文件"></a>2. 配置参数文件</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; /opt/kubernetes/cfg/kubelet-config.yml &lt;&lt; EOF</span><br><span class="line">kind: KubeletConfiguration</span><br><span class="line">apiVersion: kubelet.config.k8s.io/v1beta1</span><br><span class="line">address: 0.0.0.0</span><br><span class="line">port: 10250</span><br><span class="line">readOnlyPort: 10255</span><br><span class="line">cgroupDriver: cgroupfs</span><br><span class="line">clusterDNS:</span><br><span class="line">- 10.0.0.2</span><br><span class="line">clusterDomain: cluster.local </span><br><span class="line">failSwapOn: false</span><br><span class="line">authentication:</span><br><span class="line">  anonymous:</span><br><span class="line">    enabled: false</span><br><span class="line">  webhook:</span><br><span class="line">    cacheTTL: 2m0s</span><br><span class="line">    enabled: true</span><br><span class="line">  x509:</span><br><span class="line">    clientCAFile: /opt/kubernetes/ssl/ca.pem </span><br><span class="line">authorization:</span><br><span class="line">  mode: Webhook</span><br><span class="line">  webhook:</span><br><span class="line">    cacheAuthorizedTTL: 5m0s</span><br><span class="line">    cacheUnauthorizedTTL: 30s</span><br><span class="line">evictionHard:</span><br><span class="line">  imagefs.available: 15%</span><br><span class="line">  memory.available: 100Mi</span><br><span class="line">  nodefs.available: 10%</span><br><span class="line">  nodefs.inodesFree: 5%</span><br><span class="line">maxOpenFiles: 1000000</span><br><span class="line">maxPods: 110</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>
<h3 id="3-生成kubelet初次加入集群引导kubeconfig文件【master中执行，把得到的文件传到Node中】"><a href="#3-生成kubelet初次加入集群引导kubeconfig文件【master中执行，把得到的文件传到Node中】" class="headerlink" title="3. 生成kubelet初次加入集群引导kubeconfig文件【master中执行，把得到的文件传到Node中】"></a>3. 生成kubelet初次加入集群引导kubeconfig文件【master中执行，把得到的文件传到Node中】</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">KUBE_CONFIG=&quot;/opt/kubernetes/cfg/bootstrap.kubeconfig&quot;</span><br><span class="line">KUBE_APISERVER=&quot;https://192.168.100.61:6443&quot; # apiserver IP:PORT</span><br><span class="line">TOKEN=&quot;c47ffb939f5ca36231d9e3121a252940&quot; # 与token.csv里保持一致 master中存储位置/opt/kubernetes/cfg/token.csv</span><br><span class="line"></span><br><span class="line"># 生成 kubelet bootstrap kubeconfig 配置文件</span><br><span class="line">kubectl config set-cluster kubernetes \</span><br><span class="line">  --certificate-authority=/opt/kubernetes/ssl/ca.pem \</span><br><span class="line">  --embed-certs=true \</span><br><span class="line">  --server=$&#123;KUBE_APISERVER&#125; \</span><br><span class="line">  --kubeconfig=$&#123;KUBE_CONFIG&#125;</span><br><span class="line">kubectl config set-credentials &quot;kubelet-bootstrap&quot; \</span><br><span class="line">  --token=$&#123;TOKEN&#125; \</span><br><span class="line">  --kubeconfig=$&#123;KUBE_CONFIG&#125;</span><br><span class="line">kubectl config set-context default \</span><br><span class="line">  --cluster=kubernetes \</span><br><span class="line">  --user=&quot;kubelet-bootstrap&quot; \</span><br><span class="line">  --kubeconfig=$&#123;KUBE_CONFIG&#125;</span><br><span class="line">kubectl config use-context default --kubeconfig=$&#123;KUBE_CONFIG&#125;</span><br></pre></td></tr></table></figure>
<p><img src="https://img-blog.csdnimg.cn/ae18f0a6873a4f00a7a5a3ed6c7585e1.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5rKn5rW35LmL5beF,size_20,color_FFFFFF,t_70,g_se,x_16" alt="在这里插入图片描述"></p>
<h3 id="4-systemd管理kubelet【Node】"><a href="#4-systemd管理kubelet【Node】" class="headerlink" title="4. systemd管理kubelet【Node】"></a>4. systemd管理kubelet【Node】</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; /usr/lib/systemd/system/kubelet.service &lt;&lt; EOF</span><br><span class="line">[Unit]</span><br><span class="line">Description=Kubernetes Kubelet</span><br><span class="line">After=docker.service</span><br><span class="line"></span><br><span class="line">[Service]</span><br><span class="line">EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf</span><br><span class="line">ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS</span><br><span class="line">Restart=on-failure</span><br><span class="line">LimitNOFILE=65536</span><br><span class="line"></span><br><span class="line">[Install]</span><br><span class="line">WantedBy=multi-user.target</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>

<h3 id="5-启动并设置开机启动【Node】"><a href="#5-启动并设置开机启动【Node】" class="headerlink" title="5. 启动并设置开机启动【Node】"></a>5. 启动并设置开机启动【Node】</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">[root@k8s-node1 cfg]# systemctl daemon-reload</span><br><span class="line">[root@k8s-node1 cfg]# systemctl start kubelet &amp;&amp; systemctl enable kubelet</span><br><span class="line">[root@k8s-node1 cfg]# systemctl status kubelet</span><br><span class="line">● kubelet.service - Kubernetes Kubelet</span><br><span class="line">   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)</span><br><span class="line">   Active: active (running) since Mon 2021-12-13 16:44:45 PST; 9s ago</span><br><span class="line"> Main PID: 39675 (kubelet)</span><br><span class="line">    Tasks: 7</span><br><span class="line">   Memory: 114.0M</span><br><span class="line">   CGroup: /system.slice/kubelet.service</span><br><span class="line">           └─39675 /opt/kubernetes/bin/kubelet --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --hostname-override=k8s-master1 --network-plugin=cni --kubeconfig...</span><br><span class="line"></span><br><span class="line">Dec 13 16:44:45 k8s-node1 systemd[1]: Started Kubernetes Kubelet.</span><br><span class="line">[root@k8s-node1 cfg]#</span><br></pre></td></tr></table></figure>
<p><img src="https://img-blog.csdnimg.cn/527cf3121ca74111ba53dc1444e3618b.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5rKn5rW35LmL5beF,size_20,color_FFFFFF,t_70,g_se,x_16" alt="在这里插入图片描述"></p>
<h2 id="5-3-批准kubelet证书申请并加入集群-Master中操作"><a href="#5-3-批准kubelet证书申请并加入集群-Master中操作" class="headerlink" title="5.3 批准kubelet证书申请并加入集群[Master中操作]"></a>5.3 批准kubelet证书申请并加入集群[Master中操作]</h2><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"># 查看kubelet证书请求</span><br><span class="line">[root@k8s-master1 cfg]# kubectl get csr</span><br><span class="line">NAME                                                   AGE   SIGNERNAME                                    REQUESTOR           CONDITION</span><br><span class="line">node-csr-TCzj1DLD2assioRywrCeVN9zM-xZdCS4MD8tFRgMAfk   51s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Pending</span><br><span class="line"></span><br><span class="line"># 批准申请</span><br><span class="line">[root@k8s-master1 cfg]#</span><br><span class="line">[root@k8s-master1 cfg]# kubectl certificate approve node-csr-TCzj1DLD2assioRywrCeVN9zM-xZdCS4MD8tFRgMAfk</span><br><span class="line">certificatesigningrequest.certificates.k8s.io/node-csr-TCzj1DLD2assioRywrCeVN9zM-xZdCS4MD8tFRgMAfk approved</span><br><span class="line"></span><br><span class="line"># 查看节点</span><br><span class="line">[root@k8s-master1 cfg]# kubectl get node</span><br><span class="line">NAME          STATUS     ROLES    AGE   VERSION</span><br><span class="line">k8s-master1   NotReady   &lt;none&gt;   24s   v1.20.13</span><br></pre></td></tr></table></figure>
<p><img src="https://img-blog.csdnimg.cn/0c451de5e0fa468884b9c8a978e9e94c.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5rKn5rW35LmL5beF,size_20,color_FFFFFF,t_70,g_se,x_16" alt="在这里插入图片描述"><br>注：由于网络插件还没有部署，节点会没有准备就绪 NotReady</p>
<h2 id="5-4-部署kube-proxy"><a href="#5-4-部署kube-proxy" class="headerlink" title="5.4 部署kube-proxy"></a>5.4 部署kube-proxy</h2><h3 id="1-创建配置文件【Node】"><a href="#1-创建配置文件【Node】" class="headerlink" title="1. 创建配置文件【Node】"></a>1. 创建配置文件【Node】</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; /opt/kubernetes/cfg/kube-proxy.conf &lt;&lt; EOF</span><br><span class="line">KUBE_PROXY_OPTS=&quot;--logtostderr=false \\</span><br><span class="line">--v=2 \\</span><br><span class="line">--log-dir=/opt/kubernetes/logs \\</span><br><span class="line">--config=/opt/kubernetes/cfg/kube-proxy-config.yml&quot;</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>
<h3 id="2-配置参数文件【Node】"><a href="#2-配置参数文件【Node】" class="headerlink" title="2. 配置参数文件【Node】"></a>2. 配置参数文件【Node】</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; /opt/kubernetes/cfg/kube-proxy-config.yml &lt;&lt; EOF</span><br><span class="line">kind: KubeProxyConfiguration</span><br><span class="line">apiVersion: kubeproxy.config.k8s.io/v1alpha1</span><br><span class="line">bindAddress: 0.0.0.0</span><br><span class="line">metricsBindAddress: 0.0.0.0:10249</span><br><span class="line">clientConnection:</span><br><span class="line">  kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig</span><br><span class="line">hostnameOverride: k8s-master1</span><br><span class="line">clusterCIDR: 10.244.0.0/16</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>
<h3 id="3-生成kube-proxy-kubeconfig文件【Master】"><a href="#3-生成kube-proxy-kubeconfig文件【Master】" class="headerlink" title="3. 生成kube-proxy.kubeconfig文件【Master】"></a>3. 生成kube-proxy.kubeconfig文件【Master】</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"># 切换工作目录</span><br><span class="line">cd ~/TLS/k8s</span><br><span class="line"></span><br><span class="line"># 创建证书请求文件</span><br><span class="line">cat &gt; kube-proxy-csr.json &lt;&lt; EOF</span><br><span class="line">&#123;</span><br><span class="line">  &quot;CN&quot;: &quot;system:kube-proxy&quot;,</span><br><span class="line">  &quot;hosts&quot;: [],</span><br><span class="line">  &quot;key&quot;: &#123;</span><br><span class="line">    &quot;algo&quot;: &quot;rsa&quot;,</span><br><span class="line">    &quot;size&quot;: 2048</span><br><span class="line">  &#125;,</span><br><span class="line">  &quot;names&quot;: [</span><br><span class="line">    &#123;</span><br><span class="line">      &quot;C&quot;: &quot;CN&quot;,</span><br><span class="line">      &quot;L&quot;: &quot;BeiJing&quot;,</span><br><span class="line">      &quot;ST&quot;: &quot;BeiJing&quot;,</span><br><span class="line">      &quot;O&quot;: &quot;k8s&quot;,</span><br><span class="line">      &quot;OU&quot;: &quot;System&quot;</span><br><span class="line">    &#125;</span><br><span class="line">  ]</span><br><span class="line">&#125;</span><br><span class="line">EOF</span><br><span class="line"></span><br><span class="line"># 生成证书</span><br><span class="line">cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy</span><br></pre></td></tr></table></figure>
<p>生成kubeconfig文件：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">KUBE_CONFIG=&quot;/opt/kubernetes/cfg/kube-proxy.kubeconfig&quot;</span><br><span class="line">KUBE_APISERVER=&quot;https://192.168.100.61:6443&quot;</span><br><span class="line"></span><br><span class="line">kubectl config set-cluster kubernetes \</span><br><span class="line">  --certificate-authority=/opt/kubernetes/ssl/ca.pem \</span><br><span class="line">  --embed-certs=true \</span><br><span class="line">  --server=$&#123;KUBE_APISERVER&#125; \</span><br><span class="line">  --kubeconfig=$&#123;KUBE_CONFIG&#125;</span><br><span class="line">kubectl config set-credentials kube-proxy \</span><br><span class="line">  --client-certificate=./kube-proxy.pem \</span><br><span class="line">  --client-key=./kube-proxy-key.pem \</span><br><span class="line">  --embed-certs=true \</span><br><span class="line">  --kubeconfig=$&#123;KUBE_CONFIG&#125;</span><br><span class="line">kubectl config set-context default \</span><br><span class="line">  --cluster=kubernetes \</span><br><span class="line">  --user=kube-proxy \</span><br><span class="line">  --kubeconfig=$&#123;KUBE_CONFIG&#125;</span><br><span class="line">kubectl config use-context default --kubeconfig=$&#123;KUBE_CONFIG&#125;</span><br></pre></td></tr></table></figure>

<p>把/opt/kubernetes/cfg/kube-proxy.kubeconfig传到Node节点</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[root@k8s-master1 k8s]# scp /opt/kubernetes/cfg/kube-proxy.kubeconfig root@192.168.100.62:/opt/kubernetes/cfg/</span><br><span class="line">root@192.168.100.62&#x27;s password:</span><br><span class="line">kube-proxy.kubeconfig   </span><br></pre></td></tr></table></figure>
<h3 id="4-systemd管理kube-proxy【Node】"><a href="#4-systemd管理kube-proxy【Node】" class="headerlink" title="4. systemd管理kube-proxy【Node】"></a>4. systemd管理kube-proxy【Node】</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; /usr/lib/systemd/system/kube-proxy.service &lt;&lt; EOF</span><br><span class="line">[Unit]</span><br><span class="line">Description=Kubernetes Proxy</span><br><span class="line">After=network.target</span><br><span class="line"></span><br><span class="line">[Service]</span><br><span class="line">EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf</span><br><span class="line">ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS</span><br><span class="line">Restart=on-failure</span><br><span class="line">LimitNOFILE=65536</span><br><span class="line"></span><br><span class="line">[Install]</span><br><span class="line">WantedBy=multi-user.target</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>
<h3 id="5-启动并设置开机启动【Node】-1"><a href="#5-启动并设置开机启动【Node】-1" class="headerlink" title="5. 启动并设置开机启动【Node】"></a>5. 启动并设置开机启动【Node】</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line">systemctl daemon-reload &amp;&amp; systemctl start kube-proxy &amp;&amp; systemctl enable kube-proxy</span><br><span class="line"></span><br><span class="line">[root@k8s-node1 cfg]# systemctl daemon-reload &amp;&amp; systemctl start kube-proxy &amp;&amp; systemctl enable kube-proxy</span><br><span class="line">Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.</span><br><span class="line">[root@k8s-node1 cfg]# systemctl status kube-proxy</span><br><span class="line">● kube-proxy.service - Kubernetes Proxy</span><br><span class="line">   Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)</span><br><span class="line">   Active: active (running) since Mon 2021-12-13 17:00:35 PST; 9s ago</span><br><span class="line"> Main PID: 42943 (kube-proxy)</span><br><span class="line">    Tasks: 6</span><br><span class="line">   Memory: 47.9M</span><br><span class="line">   CGroup: /system.slice/kube-proxy.service</span><br><span class="line">           └─42943 /opt/kubernetes/bin/kube-proxy --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --config=/opt/kubernetes/cfg/kube-proxy-config.yml</span><br><span class="line"></span><br><span class="line">Dec 13 17:00:35 k8s-node1 systemd[1]: Started Kubernetes Proxy.</span><br><span class="line">[root@k8s-node1 cfg]#</span><br></pre></td></tr></table></figure>
<p><img src="https://img-blog.csdnimg.cn/2e7f7c11eb284964a0f158a861332cec.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5rKn5rW35LmL5beF,size_20,color_FFFFFF,t_70,g_se,x_16" alt="在这里插入图片描述"></p>
<h2 id="5-5-部署网络组件【Master】"><a href="#5-5-部署网络组件【Master】" class="headerlink" title="5.5 部署网络组件【Master】"></a>5.5 部署网络组件【Master】</h2><p>Calico是一个纯三层的数据中心网络方案，是目前Kubernetes主流的网络方案。<br>部署Calico：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">wget https://docs.projectcalico.org/manifests/calico.yaml</span><br><span class="line">kubectl apply -f calico.yaml</span><br><span class="line">kubectl get pods -n kube-system</span><br></pre></td></tr></table></figure>
<p>等Calico Pod都Running，节点也会准备就绪：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">kubectl get node</span><br><span class="line">NAME         STATUS   ROLES    AGE   VERSION</span><br><span class="line">k8s-master   Ready    &lt;none&gt;   37m   v1.20.4</span><br></pre></td></tr></table></figure>

<ul>
<li><a target="_blank" rel="noopener" href="https://blog.csdn.net/qq_40806970/article/details/99296983">解决 执行kubectl xxx xxx 出现 Unable to connect to the server: net/http: TLS handshake timeout 问题</a><ul>
<li>内存太小，增加到3G即可<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line">[root@k8s-master1 k8s]# kubectl apply -f calico.yaml</span><br><span class="line">configmap/calico-config created</span><br><span class="line">customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created</span><br><span class="line">customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created</span><br><span class="line">customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created</span><br><span class="line">customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created</span><br><span class="line">customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created</span><br><span class="line">customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created</span><br><span class="line">customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created</span><br><span class="line">customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created</span><br><span class="line">customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created</span><br><span class="line">customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created</span><br><span class="line">customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created</span><br><span class="line">customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created</span><br><span class="line">customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created</span><br><span class="line">customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created</span><br><span class="line">customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created</span><br><span class="line">clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created</span><br><span class="line">clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created</span><br><span class="line">clusterrole.rbac.authorization.k8s.io/calico-node created</span><br><span class="line">clusterrolebinding.rbac.authorization.k8s.io/calico-node created</span><br><span class="line">daemonset.apps/calico-node created</span><br><span class="line">serviceaccount/calico-node created</span><br><span class="line">deployment.apps/calico-kube-controllers created</span><br><span class="line">serviceaccount/calico-kube-controllers created</span><br><span class="line">[root@k8s-master1 k8s]#</span><br><span class="line">[root@k8s-master1 k8s]# kubectl get pods -n kube-system</span><br><span class="line">NAME                                      READY   STATUS    RESTARTS   AGE</span><br><span class="line">calico-kube-controllers-97769f7c7-xpbvf   1/1     Running   0          3m58s</span><br><span class="line">calico-node-xmhw9                         1/1     Running   0          3m58s</span><br><span class="line">[root@k8s-master1 k8s]#</span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>
<img src="https://img-blog.csdnimg.cn/24bccb79829743e48580cff4421ba132.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5rKn5rW35LmL5beF,size_20,color_FFFFFF,t_70,g_se,x_16" alt="在这里插入图片描述"></li>
</ul>
</li>
</ul>
<h2 id="5-6-授权apiserver访问kubelet【Master】"><a href="#5-6-授权apiserver访问kubelet【Master】" class="headerlink" title="5.6 授权apiserver访问kubelet【Master】"></a>5.6 授权apiserver访问kubelet【Master】</h2><p>应用场景：例如kubectl logs</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; apiserver-to-kubelet-rbac.yaml &lt;&lt; EOF</span><br><span class="line">apiVersion: rbac.authorization.k8s.io/v1</span><br><span class="line">kind: ClusterRole</span><br><span class="line">metadata:</span><br><span class="line">  annotations:</span><br><span class="line">    rbac.authorization.kubernetes.io/autoupdate: &quot;true&quot;</span><br><span class="line">  labels:</span><br><span class="line">    kubernetes.io/bootstrapping: rbac-defaults</span><br><span class="line">  name: system:kube-apiserver-to-kubelet</span><br><span class="line">rules:</span><br><span class="line">  - apiGroups:</span><br><span class="line">      - &quot;&quot;</span><br><span class="line">    resources:</span><br><span class="line">      - nodes/proxy</span><br><span class="line">      - nodes/stats</span><br><span class="line">      - nodes/log</span><br><span class="line">      - nodes/spec</span><br><span class="line">      - nodes/metrics</span><br><span class="line">      - pods/log</span><br><span class="line">    verbs:</span><br><span class="line">      - &quot;*&quot;</span><br><span class="line">---</span><br><span class="line">apiVersion: rbac.authorization.k8s.io/v1</span><br><span class="line">kind: ClusterRoleBinding</span><br><span class="line">metadata:</span><br><span class="line">  name: system:kube-apiserver</span><br><span class="line">  namespace: &quot;&quot;</span><br><span class="line">roleRef:</span><br><span class="line">  apiGroup: rbac.authorization.k8s.io</span><br><span class="line">  kind: ClusterRole</span><br><span class="line">  name: system:kube-apiserver-to-kubelet</span><br><span class="line">subjects:</span><br><span class="line">  - apiGroup: rbac.authorization.k8s.io</span><br><span class="line">    kind: User</span><br><span class="line">    name: kubernetes</span><br><span class="line">EOF</span><br><span class="line"></span><br><span class="line">[root@k8s-master1 k8s]# kubectl apply -f apiserver-to-kubelet-rbac.yaml</span><br><span class="line">clusterrole.rbac.authorization.k8s.io/system:kube-apiserver-to-kubelet created</span><br><span class="line">clusterrolebinding.rbac.authorization.k8s.io/system:kube-apiserver created</span><br><span class="line">[root@k8s-master1 k8s]#</span><br><span class="line"></span><br></pre></td></tr></table></figure>

<h2 id="5-7-新增加Worker-Node"><a href="#5-7-新增加Worker-Node" class="headerlink" title="5.7 新增加Worker Node"></a>5.7 新增加Worker Node</h2><h3 id="1-拷贝已部署好的Node相关文件到新节点"><a href="#1-拷贝已部署好的Node相关文件到新节点" class="headerlink" title="1. 拷贝已部署好的Node相关文件到新节点"></a>1. 拷贝已部署好的Node相关文件到新节点</h3><p>在Master节点将Worker Node涉及文件拷贝到新节点192.168.100.62/63</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">scp -r /opt/kubernetes root@192.168.100.63:/opt/</span><br><span class="line"></span><br><span class="line">scp -r /usr/lib/systemd/system/&#123;kubelet,kube-proxy&#125;.service root@192.168.100.63:/usr/lib/systemd/system</span><br><span class="line"></span><br><span class="line">scp /opt/kubernetes/ssl/ca.pem root@192.168.100.63:/opt/kubernetes/ssl</span><br></pre></td></tr></table></figure>
<h3 id="2-删除kubelet证书和kubeconfig文件"><a href="#2-删除kubelet证书和kubeconfig文件" class="headerlink" title="2. 删除kubelet证书和kubeconfig文件"></a>2. 删除kubelet证书和kubeconfig文件</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">rm -f /opt/kubernetes/cfg/kubelet.kubeconfig </span><br><span class="line">rm -f /opt/kubernetes/ssl/kubelet*</span><br></pre></td></tr></table></figure>
<p>注：==这几个文件是证书申请审批后自动生成的，每个Node不同，必须删除==</p>
<h3 id="3-修改主机名"><a href="#3-修改主机名" class="headerlink" title="3. 修改主机名"></a>3. 修改主机名</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">vi /opt/kubernetes/cfg/kubelet.conf</span><br><span class="line">--hostname-override=k8s-node1</span><br><span class="line"></span><br><span class="line">vi /opt/kubernetes/cfg/kube-proxy-config.yml</span><br><span class="line">hostnameOverride: k8s-node1</span><br></pre></td></tr></table></figure>
<h3 id="4-启动并设置开机启动-2"><a href="#4-启动并设置开机启动-2" class="headerlink" title="4. 启动并设置开机启动"></a>4. 启动并设置开机启动</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">systemctl daemon-reload &amp;&amp; systemctl start kubelet kube-proxy &amp;&amp; systemctl enable kubelet kube-proxy</span><br></pre></td></tr></table></figure>
<h3 id="5-在Master上批准新Node-kubelet证书申请【Master】"><a href="#5-在Master上批准新Node-kubelet证书申请【Master】" class="headerlink" title="5. 在Master上批准新Node kubelet证书申请【Master】"></a>5. 在Master上批准新Node kubelet证书申请【Master】</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"># 查看证书请求</span><br><span class="line">[root@k8s-master1 k8s]# kubectl get csr</span><br><span class="line">NAME                                                   AGE    SIGNERNAME                                    REQUESTOR           CONDITION</span><br><span class="line">node-csr-TCzj1DLD2assioRywrCeVN9zM-xZdCS4MD8tFRgMAfk   116m   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Approved,Issued</span><br><span class="line">node-csr-sRYKEmpGROL8OrybXHVKu8bMjYNiGM6IieT6uYJgAV8   13s    kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Pending</span><br><span class="line">[root@k8s-master1 k8s]#</span><br><span class="line"></span><br><span class="line"># 授权请求，使用Pending标记的那个</span><br><span class="line">[root@k8s-master1 k8s]# kubectl certificate approve node-csr-sRYKEmpGROL8OrybXHVKu8bMjYNiGM6IieT6uYJgAV8</span><br><span class="line">certificatesigningrequest.certificates.k8s.io/node-csr-sRYKEmpGROL8OrybXHVKu8bMjYNiGM6IieT6uYJgAV8 approved</span><br><span class="line">[root@k8s-master1 k8s]#</span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h3 id="6-查看Node状态【Master】"><a href="#6-查看Node状态【Master】" class="headerlink" title="6. 查看Node状态【Master】"></a>6. 查看Node状态【Master】</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">kubectl get node</span><br><span class="line">NAME       STATUS   ROLES    AGE     VERSION</span><br><span class="line">k8s-master1   Ready    &lt;none&gt;   47m     v1.20.4</span><br><span class="line">k8s-node1    Ready    &lt;none&gt;   6m49s   v1.20.4</span><br></pre></td></tr></table></figure>
<p>Node2（192.168.100.63 ）节点同上。记得修改主机名！</p>
<h1 id="六、部署Dashboard和CoreDNS"><a href="#六、部署Dashboard和CoreDNS" class="headerlink" title="六、部署Dashboard和CoreDNS"></a>六、部署Dashboard和CoreDNS</h1><h2 id="6-1-部署Dashboard"><a href="#6-1-部署Dashboard" class="headerlink" title="6.1 部署Dashboard"></a>6.1 部署Dashboard</h2><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">kubectl apply -f kubernetes-dashboard.yaml</span><br><span class="line"># 查看部署</span><br><span class="line">kubectl get pods,svc -n kubernetes-dashboard</span><br></pre></td></tr></table></figure>

<p>访问地址：<a target="_blank" rel="noopener" href="https://nodeip:30001/">https://NodeIP:30001</a><br>创建service account并绑定默认cluster-admin管理员集群角色：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">kubectl create serviceaccount dashboard-admin -n kube-system</span><br><span class="line">kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin</span><br><span class="line">kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk &#x27;/dashboard-admin/&#123;print $1&#125;&#x27;)</span><br></pre></td></tr></table></figure>

<p>使用输出的token登录Dashboard。<br><img src="https://img-blog.csdnimg.cn/fc7d266de36c4502905f42c2999f4f32.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5rKn5rW35LmL5beF,size_19,color_FFFFFF,t_70,g_se,x_16" alt="在这里插入图片描述"></p>
<h2 id="6-2-部署CoreDNS"><a href="#6-2-部署CoreDNS" class="headerlink" title="6.2 部署CoreDNS"></a>6.2 部署CoreDNS</h2><p>CoreDNS用于集群内部Service名称解析。</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">kubectl apply -f coredns.yaml </span><br><span class="line"> </span><br><span class="line">kubectl get pods -n kube-system  </span><br><span class="line">NAME                          READY   STATUS    RESTARTS   AGE </span><br><span class="line">coredns-5ffbfd976d-j6shb      1/1     Running   0          32s</span><br></pre></td></tr></table></figure>
<p>DNS解析测试：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">kubectl run -it --rm dns-test --image=busybox:1.28.4 sh </span><br><span class="line">If you don&#x27;t see a command prompt, try pressing enter. </span><br><span class="line"> </span><br><span class="line">/ # nslookup kubernetes </span><br><span class="line">Server:    10.0.0.2 </span><br><span class="line">Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local </span><br><span class="line"> </span><br><span class="line">Name:      kubernetes </span><br><span class="line">Address 1: 10.0.0.1 kubernetes.default.svc.cluster.local</span><br></pre></td></tr></table></figure>
<p>解析没问题。<br>至此一个单Master集群就搭建完成了！这个环境就足以满足学习实验了，如果你的服务器配置较高，可继续扩容多Master集群！</p>
<h1 id="七、扩容多Master（高可用架构）"><a href="#七、扩容多Master（高可用架构）" class="headerlink" title="七、扩容多Master（高可用架构）"></a>七、扩容多Master（高可用架构）</h1><p>&emsp;&emsp;Kubernetes作为容器集群系统，通过健康检查+重启策略实现了Pod故障自我修复能力，通过调度算法实现将Pod分布式部署，并保持预期副本数，根据Node失效状态自动在其他Node拉起Pod，实现了应用层的高可用性。<br>&emsp;&emsp;针对Kubernetes集群，高可用性还应包含以下两个层面的考虑：Etcd数据库的高可用性和Kubernetes Master组件的高可用性。 而Etcd我们已经采用3个节点组建集群实现高可用，本节将对Master节点高可用进行说明和实施。<br>&emsp;&emsp;Master节点扮演着总控中心的角色，通过不断与工作节点上的Kubelet和kube-proxy进行通信来维护整个集群的健康工作状态。如果Master节点故障，将无法使用kubectl工具或者API做任何集群管理。<br>&emsp;&emsp;Master节点主要有三个服务kube-apiserver、kube-controller-manager和kube-scheduler，其中kube-controller-manager和kube-scheduler组件自身通过选择机制已经实现了高可用，所以Master高可用主要针对kube-apiserver组件，而该组件是以HTTP API提供服务，因此对他高可用与Web服务器类似，增加负载均衡器对其负载均衡即可，并且可水平扩容。  </p>
<p>多Master架构图：<br><img src="https://img-blog.csdnimg.cn/7693fdd716af42bb9fee3224e80a4c08.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5rKn5rW35LmL5beF,size_19,color_FFFFFF,t_70,g_se,x_16" alt="在这里插入图片描述"></p>
<h2 id="7-1-部署Master2-Node"><a href="#7-1-部署Master2-Node" class="headerlink" title="7.1 部署Master2 Node"></a>7.1 部署Master2 Node</h2><p>&emsp;&emsp;现在需要再增加一台新服务器，作为Master2 Node，IP是192.168.100.64。<br>&emsp;&emsp;为了节省资源你也可以将之前部署好的Worker Node1复用为Master2 Node角色（即部署Master组件）<br>&emsp;&emsp;Master2 与已部署的Master1所有操作一致。所以我们只需将Master1所有K8s文件拷贝过来，再修改下服务器IP和主机名启动即可。 </p>
<h3 id="1-安装Docker"><a href="#1-安装Docker" class="headerlink" title="1. 安装Docker"></a>1. 安装Docker</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">scp /usr/bin/docker* root@192.168.100.64:/usr/bin</span><br><span class="line">scp /usr/bin/runc root@192.168.100.64:/usr/bin</span><br><span class="line">scp /usr/bin/containerd* root@192.168.100.64:/usr/bin</span><br><span class="line">scp /usr/lib/systemd/system/docker.service root@192.168.100.64:/usr/lib/systemd/system</span><br><span class="line">scp -r /etc/docker root@192.168.100.64:/etc</span><br><span class="line"></span><br><span class="line"># 在Master2启动Docker</span><br><span class="line">systemctl daemon-reload</span><br><span class="line">systemctl start docker</span><br><span class="line">systemctl enable docker</span><br></pre></td></tr></table></figure>
<h3 id="2-创建etcd证书目录"><a href="#2-创建etcd证书目录" class="headerlink" title="2. 创建etcd证书目录"></a>2. 创建etcd证书目录</h3><p>在Master2创建etcd证书目录：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">mkdir -p /opt/etcd/ssl</span><br></pre></td></tr></table></figure>
<h3 id="3-拷贝文件（Master1操作）"><a href="#3-拷贝文件（Master1操作）" class="headerlink" title="3. 拷贝文件（Master1操作）"></a>3. 拷贝文件（Master1操作）</h3><p>拷贝Master1上所有K8s文件和etcd证书到Master2：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">scp -r /opt/kubernetes root@192.168.100.64:/opt</span><br><span class="line">scp -r /opt/etcd/ssl root@192.168.100.64:/opt/etcd</span><br><span class="line">scp /usr/lib/systemd/system/kube* root@192.168.100.64:/usr/lib/systemd/system</span><br><span class="line">scp /usr/bin/kubectl  root@192.168.100.64:/usr/bin</span><br><span class="line">scp -r ~/.kube root@192.168.100.64:~</span><br></pre></td></tr></table></figure>
<h3 id="4-删除证书文件"><a href="#4-删除证书文件" class="headerlink" title="4. 删除证书文件"></a>4. 删除证书文件</h3><p>删除kubelet证书和kubeconfig文件：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">rm -f /opt/kubernetes/cfg/kubelet.kubeconfig </span><br><span class="line">rm -f /opt/kubernetes/ssl/kubelet*</span><br></pre></td></tr></table></figure>
<h3 id="5-修改配置文件IP和主机名"><a href="#5-修改配置文件IP和主机名" class="headerlink" title="5. 修改配置文件IP和主机名"></a>5. 修改配置文件IP和主机名</h3><p>修改apiserver、kubelet和kube-proxy配置文件为本地IP：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line">vi /opt/kubernetes/cfg/kube-apiserver.conf </span><br><span class="line">...</span><br><span class="line">--bind-address=192.168.100.64 \</span><br><span class="line">--advertise-address=192.168.100.64 \</span><br><span class="line">...</span><br><span class="line"></span><br><span class="line">vi /opt/kubernetes/cfg/kube-controller-manager.kubeconfig</span><br><span class="line">server: https://192.168.100.64:6443</span><br><span class="line"></span><br><span class="line">vi /opt/kubernetes/cfg/kube-scheduler.kubeconfig</span><br><span class="line">server: https://192.168.100.64:6443</span><br><span class="line"></span><br><span class="line">vi /opt/kubernetes/cfg/kubelet.conf</span><br><span class="line">--hostname-override=k8s-master2</span><br><span class="line"></span><br><span class="line">vi /opt/kubernetes/cfg/kube-proxy-config.yml</span><br><span class="line">hostnameOverride: k8s-master2</span><br><span class="line"></span><br><span class="line">vi ~/.kube/config</span><br><span class="line">...</span><br><span class="line">server: https://192.168.100.64:6443</span><br></pre></td></tr></table></figure>
<h3 id="6-启动设置开机启动"><a href="#6-启动设置开机启动" class="headerlink" title="6. 启动设置开机启动"></a>6. 启动设置开机启动</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">systemctl daemon-reload</span><br><span class="line">systemctl start kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy</span><br><span class="line">systemctl enable kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy</span><br></pre></td></tr></table></figure>
<h3 id="7-查看集群状态-1"><a href="#7-查看集群状态-1" class="headerlink" title="7. 查看集群状态"></a>7. 查看集群状态</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">kubectl get cs</span><br><span class="line">NAME                STATUS    MESSAGE             ERROR</span><br><span class="line">scheduler             Healthy   ok                  </span><br><span class="line">controller-manager       Healthy   ok                  </span><br><span class="line">etcd-1               Healthy   &#123;&quot;health&quot;:&quot;true&quot;&#125;   </span><br><span class="line">etcd-2               Healthy   &#123;&quot;health&quot;:&quot;true&quot;&#125;   </span><br><span class="line">etcd-0               Healthy   &#123;&quot;health&quot;:&quot;true&quot;&#125;</span><br></pre></td></tr></table></figure>
<h3 id="8-批准kubelet证书申请"><a href="#8-批准kubelet证书申请" class="headerlink" title="8. 批准kubelet证书申请"></a>8. 批准kubelet证书申请</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"># 查看证书请求</span><br><span class="line">kubectl get csr</span><br><span class="line">NAME                      AGE          SIGNERNAME          REQUESTOR           CONDITION</span><br><span class="line">node-csr-JYNknakEa_YpHz797oKaN-ZTk43nD51Zc9CJkBLcASU   85m   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Pending</span><br><span class="line"># 授权请求</span><br><span class="line">kubectl certificate approve node-csr-JYNknakEa_YpHz797oKaN-ZTk43nD51Zc9CJkBLcASU</span><br><span class="line"></span><br><span class="line"># 查看Node</span><br><span class="line">kubectl get node</span><br><span class="line">NAME        STATUS   ROLES    AGE   VERSION</span><br><span class="line">k8s-master1    Ready    &lt;none&gt;   34h   v1.20.4</span><br><span class="line">k8s-master2    Ready    &lt;none&gt;   2m   v1.20.4</span><br><span class="line">k8s-node1     Ready    &lt;none&gt;   33h   v1.20.4</span><br><span class="line">k8s-node2     Ready    &lt;none&gt;   33h   v1.20.4</span><br></pre></td></tr></table></figure>
<p>如果你在学习中遇到问题或者文档有误可联系阿良~ 微信: k8init</p>
<h2 id="7-2-部署Nginx-Keepalived高可用负载均衡器"><a href="#7-2-部署Nginx-Keepalived高可用负载均衡器" class="headerlink" title="7.2 部署Nginx+Keepalived高可用负载均衡器"></a>7.2 部署Nginx+Keepalived高可用负载均衡器</h2><p>kube-apiserver高可用架构图：<br><img src="https://img-blog.csdnimg.cn/602c56c336a740cf8947717a7a5f7306.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5rKn5rW35LmL5beF,size_19,color_FFFFFF,t_70,g_se,x_16" alt="在这里插入图片描述"></p>
<blockquote>
<p>ginx是一个主流Web服务和反向代理服务器，这里用四层实现对apiserver实现负载均衡。<br>Keepalived是一个主流高可用软件，基于VIP绑定实现服务器双机热备，在上述拓扑中，Keepalived主要根据Nginx运行状态判断是否需要故障转移（漂移VIP），例如当Nginx主节点挂掉，VIP会自动绑定在Nginx备节点，从而保证VIP一直可用，实现Nginx高可用。  </p>
</blockquote>
<p>注1：为了节省机器，这里与K8s Master节点机器复用。也可以独立于k8s集群之外部署，只要nginx与apiserver能通信就行。<br>注2：如果你是在公有云上，一般都不支持keepalived，那么你可以直接用它们的负载均衡器产品，直接负载均衡多台Master kube-apiserver，架构与上面一样。<br>在两台Master节点操作。</p>
<h3 id="1-安装软件包（主-备）"><a href="#1-安装软件包（主-备）" class="headerlink" title="1. 安装软件包（主/备）"></a>1. 安装软件包（主/备）</h3> <figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">yum install epel-release -y</span><br><span class="line">yum install nginx keepalived -y</span><br></pre></td></tr></table></figure>
<h3 id="2-Nginx配置文件（主-备一样）"><a href="#2-Nginx配置文件（主-备一样）" class="headerlink" title="2. Nginx配置文件（主/备一样）"></a>2. Nginx配置文件（主/备一样）</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; /etc/nginx/nginx.conf &lt;&lt; &quot;EOF&quot;</span><br><span class="line">user nginx;</span><br><span class="line">worker_processes auto;</span><br><span class="line">error_log /var/log/nginx/error.log;</span><br><span class="line">pid /run/nginx.pid;</span><br><span class="line"></span><br><span class="line">include /usr/share/nginx/modules/*.conf;</span><br><span class="line"></span><br><span class="line">events &#123;</span><br><span class="line">    worker_connections 1024;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"># 四层负载均衡，为两台Master apiserver组件提供负载均衡</span><br><span class="line">stream &#123;</span><br><span class="line"></span><br><span class="line">    log_format  main  &#x27;$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent&#x27;;</span><br><span class="line"></span><br><span class="line">    access_log  /var/log/nginx/k8s-access.log  main;</span><br><span class="line"></span><br><span class="line">    upstream k8s-apiserver &#123;</span><br><span class="line">       server 192.168.100.61:6443;   # Master1 APISERVER IP:PORT</span><br><span class="line">       server 192.168.100.64:6443;   # Master2 APISERVER IP:PORT</span><br><span class="line">    &#125;</span><br><span class="line">    </span><br><span class="line">    server &#123;</span><br><span class="line">       listen 16443; # 由于nginx与master节点复用，这个监听端口不能是6443，否则会冲突</span><br><span class="line">       proxy_pass k8s-apiserver;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">http &#123;</span><br><span class="line">    log_format  main  &#x27;$remote_addr - $remote_user [$time_local] &quot;$request&quot; &#x27;</span><br><span class="line">                      &#x27;$status $body_bytes_sent &quot;$http_referer&quot; &#x27;</span><br><span class="line">                      &#x27;&quot;$http_user_agent&quot; &quot;$http_x_forwarded_for&quot;&#x27;;</span><br><span class="line"></span><br><span class="line">    access_log  /var/log/nginx/access.log  main;</span><br><span class="line"></span><br><span class="line">    sendfile            on;</span><br><span class="line">    tcp_nopush          on;</span><br><span class="line">    tcp_nodelay         on;</span><br><span class="line">    keepalive_timeout   65;</span><br><span class="line">    types_hash_max_size 2048;</span><br><span class="line"></span><br><span class="line">    include             /etc/nginx/mime.types;</span><br><span class="line">    default_type        application/octet-stream;</span><br><span class="line"></span><br><span class="line">    server &#123;</span><br><span class="line">        listen       80 default_server;</span><br><span class="line">        server_name  _;</span><br><span class="line"></span><br><span class="line">        location / &#123;</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>
<h3 id="3-keepalived配置文件（Nginx-Master）"><a href="#3-keepalived配置文件（Nginx-Master）" class="headerlink" title="3. keepalived配置文件（Nginx Master）"></a>3. keepalived配置文件（Nginx Master）</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; /etc/keepalived/keepalived.conf &lt;&lt; EOF</span><br><span class="line">global_defs &#123; </span><br><span class="line">   notification_email &#123; </span><br><span class="line">     acassen@firewall.loc </span><br><span class="line">     failover@firewall.loc </span><br><span class="line">     sysadmin@firewall.loc </span><br><span class="line">   &#125; </span><br><span class="line">   notification_email_from Alexandre.Cassen@firewall.loc  </span><br><span class="line">   smtp_server 127.0.0.1 </span><br><span class="line">   smtp_connect_timeout 30 </span><br><span class="line">   router_id NGINX_MASTER</span><br><span class="line">&#125; </span><br><span class="line"></span><br><span class="line">vrrp_script check_nginx &#123;</span><br><span class="line">    script &quot;/etc/keepalived/check_nginx.sh&quot;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">vrrp_instance VI_1 &#123; </span><br><span class="line">    state MASTER </span><br><span class="line">    interface ens33  # 修改为实际网卡名</span><br><span class="line">    virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的 </span><br><span class="line">    priority 100    # 优先级，备服务器设置 90 </span><br><span class="line">    advert_int 1    # 指定VRRP 心跳包通告间隔时间，默认1秒 </span><br><span class="line">    authentication &#123; </span><br><span class="line">        auth_type PASS      </span><br><span class="line">        auth_pass 1111 </span><br><span class="line">    &#125;  </span><br><span class="line">    # 虚拟IP</span><br><span class="line">    virtual_ipaddress &#123; </span><br><span class="line">        192.168.100.65/24</span><br><span class="line">    &#125; </span><br><span class="line">    track_script &#123;</span><br><span class="line">        check_nginx</span><br><span class="line">    &#125; </span><br><span class="line">&#125;</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>
<blockquote>
<p>vrrp_script：指定检查nginx工作状态脚本（根据nginx状态判断是否故障转移）<br>virtual_ipaddress：虚拟IP（VIP）<br>准备上述配置文件中检查nginx运行状态的脚本：</p>
</blockquote>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; /etc/keepalived/check_nginx.sh  &lt;&lt; &quot;EOF&quot;</span><br><span class="line">#!/bin/bash</span><br><span class="line">count=$(ss -antp |grep 16443 |egrep -cv &quot;grep|$$&quot;)</span><br><span class="line"></span><br><span class="line">if [ &quot;$count&quot; -eq 0 ];then</span><br><span class="line">    exit 1</span><br><span class="line">else</span><br><span class="line">    exit 0</span><br><span class="line">fi</span><br><span class="line">EOF</span><br><span class="line">chmod +x /etc/keepalived/check_nginx.sh</span><br><span class="line">4. keepalived配置文件（Nginx Backup）</span><br><span class="line">cat &gt; /etc/keepalived/keepalived.conf &lt;&lt; EOF</span><br><span class="line">global_defs &#123; </span><br><span class="line">   notification_email &#123; </span><br><span class="line">     acassen@firewall.loc </span><br><span class="line">     failover@firewall.loc </span><br><span class="line">     sysadmin@firewall.loc </span><br><span class="line">   &#125; </span><br><span class="line">   notification_email_from Alexandre.Cassen@firewall.loc  </span><br><span class="line">   smtp_server 127.0.0.1 </span><br><span class="line">   smtp_connect_timeout 30 </span><br><span class="line">   router_id NGINX_BACKUP</span><br><span class="line">&#125; </span><br><span class="line"></span><br><span class="line">vrrp_script check_nginx &#123;</span><br><span class="line">    script &quot;/etc/keepalived/check_nginx.sh&quot;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">vrrp_instance VI_1 &#123; </span><br><span class="line">    state BACKUP </span><br><span class="line">    interface ens33</span><br><span class="line">    virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的 </span><br><span class="line">    priority 90</span><br><span class="line">    advert_int 1</span><br><span class="line">    authentication &#123; </span><br><span class="line">        auth_type PASS      </span><br><span class="line">        auth_pass 1111 </span><br><span class="line">    &#125;  </span><br><span class="line">    virtual_ipaddress &#123; </span><br><span class="line">        192.168.100.65/24</span><br><span class="line">    &#125; </span><br><span class="line">    track_script &#123;</span><br><span class="line">        check_nginx</span><br><span class="line">    &#125; </span><br><span class="line">&#125;</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>
<p>准备上述配置文件中检查nginx运行状态的脚本：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; /etc/keepalived/check_nginx.sh  &lt;&lt; &quot;EOF&quot;</span><br><span class="line">#!/bin/bash</span><br><span class="line">count=$(ss -antp |grep 16443 |egrep -cv &quot;grep|$$&quot;)</span><br><span class="line"></span><br><span class="line">if [ &quot;$count&quot; -eq 0 ];then</span><br><span class="line">    exit 1</span><br><span class="line">else</span><br><span class="line">    exit 0</span><br><span class="line">fi</span><br><span class="line">EOF</span><br><span class="line">chmod +x /etc/keepalived/check_nginx.sh</span><br></pre></td></tr></table></figure>
<p>注：keepalived根据脚本返回状态码（0为工作正常，非0不正常）判断是否故障转移。</p>
<h3 id="5-启动并设置开机启动-2"><a href="#5-启动并设置开机启动-2" class="headerlink" title="5. 启动并设置开机启动"></a>5. 启动并设置开机启动</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">systemctl daemon-reload</span><br><span class="line">systemctl start nginx keepalived</span><br><span class="line">systemctl enable nginx keepalived</span><br></pre></td></tr></table></figure>
<h3 id="6-查看keepalived工作状态"><a href="#6-查看keepalived工作状态" class="headerlink" title="6. 查看keepalived工作状态"></a>6. 查看keepalived工作状态</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">ip addr</span><br><span class="line">1: lo: &lt;LOOPBACK,UP,LOWER_UP&gt; mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000</span><br><span class="line">    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00</span><br><span class="line">    inet 127.0.0.1/8 scope host lo</span><br><span class="line">       valid_lft forever preferred_lft forever</span><br><span class="line">    inet6 ::1/128 scope host </span><br><span class="line">       valid_lft forever preferred_lft forever</span><br><span class="line">2: ens33: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc pfifo_fast state UP group default qlen 1000</span><br><span class="line">    link/ether 00:0c:29:04:f7:2c brd ff:ff:ff:ff:ff:ff</span><br><span class="line">    inet 192.168.31.80/24 brd 192.168.31.255 scope global noprefixroute ens33</span><br><span class="line">       valid_lft forever preferred_lft forever</span><br><span class="line">    inet 192.168.100.65/24 scope global secondary ens33</span><br><span class="line">       valid_lft forever preferred_lft forever</span><br><span class="line">    inet6 fe80::20c:29ff:fe04:f72c/64 scope link </span><br><span class="line">       valid_lft forever preferred_lft forever</span><br></pre></td></tr></table></figure>
<p>可以看到，在ens33网卡绑定了192.168.100.65 虚拟IP，说明工作正常。</p>
<h3 id="7-Nginx-Keepalived高可用测试"><a href="#7-Nginx-Keepalived高可用测试" class="headerlink" title="7. Nginx+Keepalived高可用测试"></a>7. Nginx+Keepalived高可用测试</h3><p>关闭主节点Nginx，测试VIP是否漂移到备节点服务器。<br>在Nginx Master执行 pkill nginx;<br>在Nginx Backup，ip addr命令查看已成功绑定VIP。  </p>
<h3 id="8-访问负载均衡器测试"><a href="#8-访问负载均衡器测试" class="headerlink" title="8. 访问负载均衡器测试"></a>8. 访问负载均衡器测试</h3><p>找K8s集群中任意一个节点，使用curl查看K8s版本测试，使用VIP访问：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">curl -k https://192.168.100.65:16443/version</span><br><span class="line">&#123;</span><br><span class="line">  &quot;major&quot;: &quot;1&quot;,</span><br><span class="line">  &quot;minor&quot;: &quot;20&quot;,</span><br><span class="line">  &quot;gitVersion&quot;: &quot;v1.20.4&quot;,</span><br><span class="line">  &quot;gitCommit&quot;: &quot;e87da0bd6e03ec3fea7933c4b5263d151aafd07c&quot;,</span><br><span class="line">  &quot;gitTreeState&quot;: &quot;clean&quot;,</span><br><span class="line">  &quot;buildDate&quot;: &quot;2021-02-18T16:03:00Z&quot;,</span><br><span class="line">  &quot;goVersion&quot;: &quot;go1.15.8&quot;,</span><br><span class="line">  &quot;compiler&quot;: &quot;gc&quot;,</span><br><span class="line">  &quot;platform&quot;: &quot;linux/amd64&quot;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>
<p>可以正确获取到K8s版本信息，说明负载均衡器搭建正常。该请求数据流程：curl -&gt; vip(nginx) -&gt; apiserver<br>通过查看Nginx日志也可以看到转发apiserver IP：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">tail /var/log/nginx/k8s-access.log -f</span><br><span class="line">192.168.100.61 192.168.100.61:6443 - [02/Apr/2021:19:17:57 +0800] 200 423</span><br><span class="line">192.168.100.61 192.168.100.62:6443 - [02/Apr/2021:19:18:50 +0800] 200 423</span><br></pre></td></tr></table></figure>
<p>到此还没结束，还有下面最关键的一步。</p>
<h2 id="7-3-修改所有Worker-Node连接LB-VIP"><a href="#7-3-修改所有Worker-Node连接LB-VIP" class="headerlink" title="7.3 修改所有Worker Node连接LB VIP"></a>7.3 修改所有Worker Node连接LB VIP</h2><p>&emsp;&emsp;试想下，虽然我们增加了Master2 Node和负载均衡器，但是我们是从单Master架构扩容的，也就是说目前所有的Worker Node组件连接都还是Master1 Node，如果不改为连接VIP走负载均衡器，那么Master还是单点故障。<br>&emsp;&emsp;因此接下来就是要改所有Worker Node（kubectl get node命令查看到的节点）组件配置文件，由原来192.168.100.61修改为192.168.100.65（VIP）。<br>在所有Worker Node执行：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">sed -i &#x27;s#192.168.100.61:6443#192.168.100.65:16443#&#x27; /opt/kubernetes/cfg/*</span><br><span class="line">systemctl restart kubelet kube-proxy</span><br></pre></td></tr></table></figure>
<p>检查节点状态：</p>
<figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">kubectl get node </span><br><span class="line">NAME         STATUS   ROLES    AGE   VERSION </span><br><span class="line">k8s-master1   Ready    &lt;none&gt;   32d   v1.20.4 </span><br><span class="line">k8s-master2   Ready    &lt;none&gt;   10m   v1.20.4 </span><br><span class="line">k8s-node1    Ready    &lt;none&gt;   31d   v1.20.4 </span><br><span class="line">k8s-node2    Ready    &lt;none&gt;   31d   v1.20.4</span><br></pre></td></tr></table></figure>
<p>至此，一套完整的 Kubernetes 高可用集群就部署完成了！</p>
 
      <!-- reward -->
      
      <div id="reword-out">
        <div id="reward-btn">
          打赏
        </div>
      </div>
      
    </div>
    

    <!-- copyright -->
    
    <div class="declare">
      <ul class="post-copyright">
        <li>
          <i class="ri-copyright-line"></i>
          <strong>版权声明： </strong>
          
          本博客所有文章除特别声明外，著作权归作者所有。转载请注明出处！
          
        </li>
      </ul>
    </div>
    
    <footer class="article-footer">
       
<div class="share-btn">
      <span class="share-sns share-outer">
        <i class="ri-share-forward-line"></i>
        分享
      </span>
      <div class="share-wrap">
        <i class="arrow"></i>
        <div class="share-icons">
          
          <a class="weibo share-sns" href="javascript:;" data-type="weibo">
            <i class="ri-weibo-fill"></i>
          </a>
          <a class="weixin share-sns wxFab" href="javascript:;" data-type="weixin">
            <i class="ri-wechat-fill"></i>
          </a>
          <a class="qq share-sns" href="javascript:;" data-type="qq">
            <i class="ri-qq-fill"></i>
          </a>
          <a class="douban share-sns" href="javascript:;" data-type="douban">
            <i class="ri-douban-line"></i>
          </a>
          <!-- <a class="qzone share-sns" href="javascript:;" data-type="qzone">
            <i class="icon icon-qzone"></i>
          </a> -->
          
          <a class="facebook share-sns" href="javascript:;" data-type="facebook">
            <i class="ri-facebook-circle-fill"></i>
          </a>
          <a class="twitter share-sns" href="javascript:;" data-type="twitter">
            <i class="ri-twitter-fill"></i>
          </a>
          <a class="google share-sns" href="javascript:;" data-type="google">
            <i class="ri-google-fill"></i>
          </a>
        </div>
      </div>
</div>

<div class="wx-share-modal">
    <a class="modal-close" href="javascript:;"><i class="ri-close-circle-line"></i></a>
    <p>扫一扫，分享到微信</p>
    <div class="wx-qrcode">
      <img src="//api.qrserver.com/v1/create-qr-code/?size=150x150&data=http://zhaoguo2008.gitee.io/2021/11/16/%E3%80%8A%E4%BA%8C%E8%BF%9B%E5%88%B6%E6%96%B9%E5%BC%8F%E6%90%AD%E5%BB%BA%E4%B8%80%E4%B8%AA%E5%AE%8C%E6%95%B4K8s%E9%9B%86%E7%BE%A4%E3%80%8Bv1.20-%E8%AF%A6%E7%BB%86%E7%89%88/" alt="微信分享二维码">
    </div>
</div>

<div id="share-mask"></div>  
  <ul class="article-tag-list" itemprop="keywords"><li class="article-tag-list-item"><a class="article-tag-list-link" href="/tags/Kubernetes/" rel="tag">Kubernetes</a></li></ul>

    </footer>
  </div>

   
  <nav class="article-nav">
    
      <a href="/2021/11/16/Docker%E5%B8%B8%E7%94%A8%E5%91%BD%E4%BB%A4/" class="article-nav-link">
        <strong class="article-nav-caption">上一篇</strong>
        <div class="article-nav-title">
          
            Docker常用命令
          
        </div>
      </a>
    
    
      <a href="/2021/11/16/Hexo%E5%AE%89%E8%A3%85%E9%83%A8%E7%BD%B2/" class="article-nav-link">
        <strong class="article-nav-caption">下一篇</strong>
        <div class="article-nav-title">Hexo安装部署</div>
      </a>
    
  </nav>

   
<!-- valine评论 -->
<div id="vcomments-box">
  <div id="vcomments"></div>
</div>
<script src="//cdn1.lncld.net/static/js/3.0.4/av-min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/valine@1.4.14/dist/Valine.min.js"></script>
<script>
  new Valine({
    el: "#vcomments",
    app_id: "TIy1f3Lsz6asTVdGVmyLUz0L-MdYXbMMI",
    app_key: "B1lMKqlUDdHxdWnLXOB4oQkg",
    path: window.location.pathname,
    avatar: "wavatar",
    placeholder: "给我的文章加点评论吧~",
    recordIP: true,
  });
  const infoEle = document.querySelector("#vcomments .info");
  if (infoEle && infoEle.childNodes && infoEle.childNodes.length > 0) {
    infoEle.childNodes.forEach(function (item) {
      item.parentNode.removeChild(item);
    });
  }
</script>
<style>
  #vcomments-box {
    padding: 5px 30px;
  }

  @media screen and (max-width: 800px) {
    #vcomments-box {
      padding: 5px 0px;
    }
  }

  #vcomments-box #vcomments {
    background-color: #fff;
  }

  .v .vlist .vcard .vh {
    padding-right: 20px;
  }

  .v .vlist .vcard {
    padding-left: 10px;
  }
</style>

 
   
     
</article>

</section>
      <footer class="footer">
  <div class="outer">
    <ul>
      <li>
        Copyrights &copy;
        2021
        <i class="ri-heart-fill heart_icon"></i> Gray Wang
      </li>
    </ul>
    <ul>
      <li>
        
      </li>
    </ul>
    <ul>
      <li>
        
        
        <span>
  <span><i class="ri-user-3-fill"></i>访问人数:<span id="busuanzi_value_site_uv"></span></span>
  <span class="division">|</span>
  <span><i class="ri-eye-fill"></i>浏览次数:<span id="busuanzi_value_page_pv"></span></span>
</span>
        
      </li>
    </ul>
    <ul>
      
    </ul>
    <ul>
      
    </ul>
    <ul>
      <li>
        <!-- cnzz统计 -->
        
        <script type="text/javascript" src='https://s9.cnzz.com/z_stat.php?id=1278069914&amp;web_id=1278069914'></script>
        
      </li>
    </ul>
  </div>
</footer>    
    </main>
    <div class="float_btns">
      <div class="totop" id="totop">
  <i class="ri-arrow-up-line"></i>
</div>

<div class="todark" id="todark">
  <i class="ri-moon-line"></i>
</div>

    </div>
    <aside class="sidebar on">
      <button class="navbar-toggle"></button>
<nav class="navbar">
  
  <div class="logo">
    <a href="/"><img src="/images/ayer-side.svg" alt="白糖蘸一点点的空间"></a>
  </div>
  
  <ul class="nav nav-main">
    
    <li class="nav-item">
      <a class="nav-item-link" href="/">主页</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/archives">归档</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/categories">分类</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/tags">标签</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/artitalk">说说</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/friends">友链</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/2021/about">关于我</a>
    </li>
    
  </ul>
</nav>
<nav class="navbar navbar-bottom">
  <ul class="nav">
    <li class="nav-item">
      
      <a class="nav-item-link nav-item-search"  title="搜索">
        <i class="ri-search-line"></i>
      </a>
      
      
      <a class="nav-item-link" target="_blank" href="/atom.xml" title="RSS Feed">
        <i class="ri-rss-line"></i>
      </a>
      
    </li>
  </ul>
</nav>
<div class="search-form-wrap">
  <div class="local-search local-search-plugin">
  <input type="search" id="local-search-input" class="local-search-input" placeholder="Search...">
  <div id="local-search-result" class="local-search-result"></div>
</div>
</div>
    </aside>
    <div id="mask"></div>

<!-- #reward -->
<div id="reward">
  <span class="close"><i class="ri-close-line"></i></span>
  <p class="reward-p"><i class="ri-cup-line"></i>请我喝杯咖啡吧~</p>
  <div class="reward-box">
    
    <div class="reward-item">
      <img class="reward-img" src="/images/alipay.png">
      <span class="reward-type">支付宝</span>
    </div>
    
    
    <div class="reward-item">
      <img class="reward-img" src="/images/wechat.png">
      <span class="reward-type">微信</span>
    </div>
    
  </div>
</div>
    
<script src="/js/jquery-3.6.0.min.js"></script>
 
<script src="/js/lazyload.min.js"></script>

<!-- Tocbot -->
 
<script src="/js/tocbot.min.js"></script>

<script>
  tocbot.init({
    tocSelector: ".tocbot",
    contentSelector: ".article-entry",
    headingSelector: "h1, h2, h3, h4, h5, h6",
    hasInnerContainers: true,
    scrollSmooth: true,
    scrollContainer: "main",
    positionFixedSelector: ".tocbot",
    positionFixedClass: "is-position-fixed",
    fixedSidebarOffset: "auto",
  });
</script>

<script src="https://cdn.jsdelivr.net/npm/jquery-modal@0.9.2/jquery.modal.min.js"></script>
<link
  rel="stylesheet"
  href="https://cdn.jsdelivr.net/npm/jquery-modal@0.9.2/jquery.modal.min.css"
/>
<script src="https://cdn.jsdelivr.net/npm/justifiedGallery@3.7.0/dist/js/jquery.justifiedGallery.min.js"></script>

<script src="/dist/main.js"></script>

<!-- ImageViewer -->
 <!-- Root element of PhotoSwipe. Must have class pswp. -->
<div class="pswp" tabindex="-1" role="dialog" aria-hidden="true">

    <!-- Background of PhotoSwipe. 
         It's a separate element as animating opacity is faster than rgba(). -->
    <div class="pswp__bg"></div>

    <!-- Slides wrapper with overflow:hidden. -->
    <div class="pswp__scroll-wrap">

        <!-- Container that holds slides. 
            PhotoSwipe keeps only 3 of them in the DOM to save memory.
            Don't modify these 3 pswp__item elements, data is added later on. -->
        <div class="pswp__container">
            <div class="pswp__item"></div>
            <div class="pswp__item"></div>
            <div class="pswp__item"></div>
        </div>

        <!-- Default (PhotoSwipeUI_Default) interface on top of sliding area. Can be changed. -->
        <div class="pswp__ui pswp__ui--hidden">

            <div class="pswp__top-bar">

                <!--  Controls are self-explanatory. Order can be changed. -->

                <div class="pswp__counter"></div>

                <button class="pswp__button pswp__button--close" title="Close (Esc)"></button>

                <button class="pswp__button pswp__button--share" style="display:none" title="Share"></button>

                <button class="pswp__button pswp__button--fs" title="Toggle fullscreen"></button>

                <button class="pswp__button pswp__button--zoom" title="Zoom in/out"></button>

                <!-- Preloader demo http://codepen.io/dimsemenov/pen/yyBWoR -->
                <!-- element will get class pswp__preloader--active when preloader is running -->
                <div class="pswp__preloader">
                    <div class="pswp__preloader__icn">
                        <div class="pswp__preloader__cut">
                            <div class="pswp__preloader__donut"></div>
                        </div>
                    </div>
                </div>
            </div>

            <div class="pswp__share-modal pswp__share-modal--hidden pswp__single-tap">
                <div class="pswp__share-tooltip"></div>
            </div>

            <button class="pswp__button pswp__button--arrow--left" title="Previous (arrow left)">
            </button>

            <button class="pswp__button pswp__button--arrow--right" title="Next (arrow right)">
            </button>

            <div class="pswp__caption">
                <div class="pswp__caption__center"></div>
            </div>

        </div>

    </div>

</div>

<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe.min.css">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/default-skin/default-skin.min.css">
<script src="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe-ui-default.min.js"></script>

<script>
    function viewer_init() {
        let pswpElement = document.querySelectorAll('.pswp')[0];
        let $imgArr = document.querySelectorAll(('.article-entry img:not(.reward-img)'))

        $imgArr.forEach(($em, i) => {
            $em.onclick = () => {
                // slider展开状态
                // todo: 这样不好，后面改成状态
                if (document.querySelector('.left-col.show')) return
                let items = []
                $imgArr.forEach(($em2, i2) => {
                    let img = $em2.getAttribute('data-idx', i2)
                    let src = $em2.getAttribute('data-target') || $em2.getAttribute('src')
                    let title = $em2.getAttribute('alt')
                    // 获得原图尺寸
                    const image = new Image()
                    image.src = src
                    items.push({
                        src: src,
                        w: image.width || $em2.width,
                        h: image.height || $em2.height,
                        title: title
                    })
                })
                var gallery = new PhotoSwipe(pswpElement, PhotoSwipeUI_Default, items, {
                    index: parseInt(i)
                });
                gallery.init()
            }
        })
    }
    viewer_init()
</script> 
<!-- MathJax -->

<!-- Katex -->

<!-- busuanzi  -->
 
<script src="/js/busuanzi-2.3.pure.min.js"></script>
 
<!-- ClickLove -->

<!-- ClickBoom1 -->

<!-- ClickBoom2 -->

<!-- CodeCopy -->
 
<link rel="stylesheet" href="/css/clipboard.css">
 <script src="https://cdn.jsdelivr.net/npm/clipboard@2/dist/clipboard.min.js"></script>
<script>
  function wait(callback, seconds) {
    var timelag = null;
    timelag = window.setTimeout(callback, seconds);
  }
  !function (e, t, a) {
    var initCopyCode = function(){
      var copyHtml = '';
      copyHtml += '<button class="btn-copy" data-clipboard-snippet="">';
      copyHtml += '<i class="ri-file-copy-2-line"></i><span>COPY</span>';
      copyHtml += '</button>';
      $(".highlight .code pre").before(copyHtml);
      $(".article pre code").before(copyHtml);
      var clipboard = new ClipboardJS('.btn-copy', {
        target: function(trigger) {
          return trigger.nextElementSibling;
        }
      });
      clipboard.on('success', function(e) {
        let $btn = $(e.trigger);
        $btn.addClass('copied');
        let $icon = $($btn.find('i'));
        $icon.removeClass('ri-file-copy-2-line');
        $icon.addClass('ri-checkbox-circle-line');
        let $span = $($btn.find('span'));
        $span[0].innerText = 'COPIED';
        
        wait(function () { // 等待两秒钟后恢复
          $icon.removeClass('ri-checkbox-circle-line');
          $icon.addClass('ri-file-copy-2-line');
          $span[0].innerText = 'COPY';
        }, 2000);
      });
      clipboard.on('error', function(e) {
        e.clearSelection();
        let $btn = $(e.trigger);
        $btn.addClass('copy-failed');
        let $icon = $($btn.find('i'));
        $icon.removeClass('ri-file-copy-2-line');
        $icon.addClass('ri-time-line');
        let $span = $($btn.find('span'));
        $span[0].innerText = 'COPY FAILED';
        
        wait(function () { // 等待两秒钟后恢复
          $icon.removeClass('ri-time-line');
          $icon.addClass('ri-file-copy-2-line');
          $span[0].innerText = 'COPY';
        }, 2000);
      });
    }
    initCopyCode();
  }(window, document);
</script>
 
<!-- CanvasBackground -->

<script>
  if (window.mermaid) {
    mermaid.initialize({ theme: "forest" });
  }
</script>


    
    

  </div>
<script src="/live2dw/lib/L2Dwidget.min.js?094cbace49a39548bed64abff5988b05"></script><script>L2Dwidget.init({"pluginRootPath":"live2dw/","pluginJsPath":"lib/","pluginModelPath":"assets/","tagMode":false,"debug":false,"model":{"jsonPath":"/live2dw/assets/haruto.model.json"},"display":{"position":"right","width":150,"height":300},"mobile":{"show":"fasle"},"log":false});</script></body>

</html>